Grid Control 11.1.0.1 with 11.2.0.2 repository and SQL net security

Grid Control 11.1.0.1 with 11.2.0.2 repository has issue with SQL net security. You might see following message/errors logged to the repository alert log very frequently. You might see these error popping up in the target database alert log files as well.

***********************************************************************

Fatal NI connect error 12170.

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.2.0 – Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.2.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.2.0 – Production
Time: 03-AUG-2011 13:44:57
Tracing not turned on.
Tns error struct:
ns main err code: 12535

TNS-12535: TNS:operation timed out
ns secondary err code: 12560
nt main err code: 505

TNS-00505: Operation timed out
nt secondary err code: 110
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.62.119.83)(PORT=4229))
Wed Aug 03 13:53:30 2011

***********************************************************************

NI cryptographic checksum mismatch error: 12599.

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.2.0 – Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.2.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.2.0 – Production
Time: 03-AUG-2011 13:53:30
Tracing not turned on.
Tns error struct:
ns main err code: 12599

TNS-12599: TNS:cryptographic checksum mismatch
ns secondary err code: 2526
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0

Along with this error there could be thousands of trace files generated in the trace directory.

This error is occurring due to encryption between the server and the client; the dispatchers are continuously dumping the following errors if the server is configured to work with some of the encryption algorithms.

*** 2010-06-02 11:01:38.188
async error encountered when accepting new connection:
NS Primary Error: TNS-12650: No common encryption or data integrity algorithm

*** 2010-06-02 11:01:40.190
async error encountered when accepting new connection:
NS Primary Error: TNS-12650: No common encryption or data integrity algorithm

*** 2010-06-02 11:01:40.190
async error encountered when accepting new connection:
NS Primary Error: TNS-12650: No common encryption or data integrity algorithm

In my case, the default Grid control installation, the server is configured to work with the AES128 encryption algorithm:

SQLNET.ENCRYPTION_TYPES_SERVER = (AES128)

If parameter SQLNET.ENCRYPTION_TYPES_SERVER is not set then the dispatchers are not dumping any error. If I comment #SQLNET.ENCRYPTION_SERVER=REQUIRED in the sqlnet.ora file the error does not happen.

The default value for SQLNET.ENCRYPTION_SERVER is ACCEPTED. So that might be the reason for accepting the connection even if there is not matching encryption algorithm.

Main reason for lot of log entries for this issue is due to the Bug 9953045. The OMS is using 10.2 JDBC Thin client version to connect to the 11.2 DB repository. By Default, the AES256 (But it happens with AES128 as well) encryption algorithm is used. But the 10.2 JDBC thin client does not support AES encryption, hence the error.

Oracle has 11.1.0.1.4 Grid Control Patch Set Update (PSU) [ID 1330064.1] to fix this issue. Here am trying to apply the grid control patch to my Grid control home.

My OMS version is

$ emctl getversion oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Enterprise Manager 11g OMS Version 11.1.0.1.0

This PSU contains three different Patches

  • Patch 12423703 for the Oracle Management Server (OMS) Oracle Home. This patch is generic for all platforms.
  • Patch 9345913 for the Oracle Management Agent Oracle Home running on Unix platforms.
  • Patch 12423714 for the Oracle Management Agent Oracle Home running on Windows platforms.

Since my Grid server is linux x86_64, I am going to do the first two Patches. In order to proceed with these patches we need patch 12620174 applied.

Steps to apply Patch 12620174

1. Perquisites: Make sure opatch,unzip are in path. Also ensure you are using latest OPatch version.

2. Copy the patch to the server and unzip

unzip p12620174_111010_Generic.zip

3. Shut down services running from the ORACLE_HOME.

$ emctl stop oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Stopping WebTier…
WebTier Successfully Stopped
Stopping Oracle Management Server…
Oracle Management Server Successfully Stopped
Oracle Management Server is Down

$ $AGENT_HOME/bin/emctl stop agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Stopping agent … stopped.

Note: Leave EM Repository database and its listener running. In case of multi-OMS environment refer README.txt

4. Set your current directory to the directory where the patch is located:

$ cd 12620174

$ opatch napply

Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation. All rights reserved.

UTIL session

Oracle Home : /opt/app/oracle/Middleware/WebLogic/oms11g
Central Inventory : /opt/app/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.2.0.1.1
OUI version : 11.1.0.8.0
OUI location : /opt/app/oracle/Middleware/WebLogic/oms11g/oui
Log file location : /opt/app/oracle/Middleware/WebLogic/oms11g/cfgtoollogs/opatch/opatch2011-08-04_10-06-52AM.log

Patch history file: /opt/app/oracle/Middleware/WebLogic/oms11g/cfgtoollogs/opatch/opatch_history.txt

OPatch detects the Middleware Home as “/opt/app/oracle/Middleware/WebLogic”

Invoking utility “napply”
Checking conflict among patches…
Checking if Oracle Home has components required by patches…
Checking conflicts against Oracle Home…
OPatch continues with these patches: 10154264 12612624 12620174

Do you want to proceed? [y|n]
y
User Responded with: Y

Running prerequisite checks…

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Backing up files affected by the patch ‘NApply’ for restore. This might take a while…

Applying patch 10154264…

ApplySession applying interim patch ‘10154264’ to OH ‘/opt/app/oracle/Middleware/WebLogic/oms11g’
Backing up files affected by the patch ‘10154264’ for rollback. This might take a while…

Patching component oracle.sysman.oms.core, 11.1.0.1.0…
Copying file to “/opt/app/oracle/Middleware/WebLogic/oms11g/sysman/admin/emdrep/sql/core/latest/admin/admin_sys_procs.sql”
Copying file to “/opt/app/oracle/Middleware/WebLogic/oms11g/sysman/admin/emdrep/sql/core/latest/admin/admin_grants_repos_user.sql”
ApplySession adding interim patch ‘10154264’ to inventory

Verifying the update…
Inventory check OK: Patch ID 10154264 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 10154264 are present in Oracle Home.

Applying patch 12612624…

ApplySession applying interim patch ‘12612624’ to OH ‘/opt/app/oracle/Middleware/WebLogic/oms11g’
Backing up files affected by the patch ‘12612624’ for rollback. This might take a while…

Patching component oracle.sysman.oms.core, 11.1.0.1.0…
Updating jar file “/opt/app/oracle/Middleware/WebLogic/oms11g/jlib/rcucommon.jar” with “/jlib/rcucommon.jar/oracle/sysman/assistants/common/dbutil/jdbc/DatabaseObjectCreationException.class”
Updating jar file “/opt/app/oracle/Middleware/WebLogic/oms11g/jlib/rcucommon.jar” with “/jlib/rcucommon.jar/oracle/sysman/assistants/common/dbutil/jdbc/DatabaseObjectCreationException$ErrorInfo.class”
Updating jar file “/opt/app/oracle/Middleware/WebLogic/oms11g/jlib/rcucommon.jar” with “/jlib/rcucommon.jar/oracle/sysman/assistants/common/dbutil/jdbc/JDBCEngine.class”
Updating jar file “/opt/app/oracle/Middleware/WebLogic/oms11g/jlib/rcucommon.jar” with “/jlib/rcucommon.jar/oracle/sysman/assistants/common/rsc/CommonMsg.class”
Updating jar file “/opt/app/oracle/Middleware/WebLogic/oms11g/jlib/rcucommon.jar” with “/jlib/rcucommon.jar/oracle/sysman/assistants/common/rsc/CommonMsgID.class”
Updating jar file “/opt/app/oracle/Middleware/WebLogic/oms11g/jlib/rcucommon.jar” with “/jlib/rcucommon.jar/oracle/sysman/assistants/common/dbutil/jdbc/DbmsOutput.class”
Updating jar file “/opt/app/oracle/Middleware/WebLogic/oms11g/jlib/rcucommon.jar” with “/jlib/rcucommon.jar/oracle/sysman/assistants/common/util/LoggingManager.class”
Copying file to “/opt/app/oracle/Middleware/WebLogic/oms11g/bin/rcuJDBCEngine”
Copying file to “/opt/app/oracle/Middleware/WebLogic/oms11g/bin/rcuJDBCEngine.bat”
ApplySession adding interim patch ‘12612624’ to inventory

Verifying the update…
Inventory check OK: Patch ID 12612624 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 12612624 are present in Oracle Home.

Applying patch 12620174…

ApplySession applying interim patch ‘12620174’ to OH ‘/opt/app/oracle/Middleware/WebLogic/oms11g’
Backing up files affected by the patch ‘12620174’ for rollback. This might take a while…

Patching component oracle.sysman.oms.core, 11.1.0.1.0…
Copying file to “/opt/app/oracle/Middleware/WebLogic/oms11g/sysman/admin/rollback_12620174.lst”
ApplySession adding interim patch ‘12620174’ to inventory

Verifying the update…
Inventory check OK: Patch ID 12620174 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 12620174 are present in Oracle Home.

The local system has been patched and can be restarted.

UtilSession: N-Apply done.

OPatch succeeded.

5. Connect to rcuJDBCEngine as SYS and execute the following sql file. Please make sure you set ORACLE_HOME to OMS_HOME before you connect to rcuJDBCEngine.

/opt/app/oracle/Middleware/WebLogic/oms11g/bin/rcuJDBCEngine sys/@11.11.11.111:1521:GCREPO JDBC_SCRIPT 10154264/patch_10154264.sql $PWD $ORACLE_HOME
Completed SQL script execution normally.
1 scripts were processed

6. Start OMS using the following command. In case of multi-OMS environment, start on all OMS machines

$ emctl start oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Starting WebTier…
WebTier Successfully Started
Starting Oracle Management Server…
Oracle Management Server Successfully Started
Oracle Management Server is Up

Steps to apply 12423703

1. Perquisites: Make sure opatch,unzip are in path. Also ensure you are using latest OPatch version.
Set ORACLE_HOME to OMS home

2. Ensure that the PSU does not conflict with the already-installed one-off patches. To do so, run the following command to generate a report that lists all conflicting patches.

$ opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./12423703
Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation. All rights reserved.

PREREQ session

Oracle Home : /opt/app/oracle/Middleware/WebLogic/oms11g
Central Inventory : /opt/app/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.2.0.1.1
OUI version : 11.1.0.8.0
OUI location : /opt/app/oracle/Middleware/WebLogic/oms11g/oui
Log file location : /opt/app/oracle/Middleware/WebLogic/oms11g/cfgtoollogs/opatch/opatch2011-08-04_11-06-55AM.log

Patch history file: /opt/app/oracle/Middleware/WebLogic/oms11g/cfgtoollogs/opatch/opatch_history.txt

OPatch detects the Middleware Home as “/opt/app/oracle/Middleware/WebLogic”

Invoking prereq “checkconflictagainstohwithdetail”

Prereq “checkConflictAgainstOHWithDetail” passed.

OPatch succeeded.

Note: If you do not see any conflicting patches refer README.txt

3. Stop all om services

emctl stop oms -all
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Stopping WebTier…
WebTier Successfully Stopped
Stopping Oracle Management Server…
Oracle Management Server Successfully Stopped
Oracle Management Server is Down

4. Unzip the p12423703_111010_Generic.zip and cd to 12423703

cd 12423703

$ opatch apply
Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation. All rights reserved.

Oracle Home : /opt/app/oracle/Middleware/WebLogic/oms11g
Central Inventory : /opt/app/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.2.0.1.1
OUI version : 11.1.0.8.0
OUI location : /opt/app/oracle/Middleware/WebLogic/oms11g/oui
Log file location : /opt/app/oracle/Middleware/WebLogic/oms11g/cfgtoollogs/opatch/opatch2011-08-04_11-11-03AM.log

Patch history file: /opt/app/oracle/Middleware/WebLogic/oms11g/cfgtoollogs/opatch/opatch_history.txt

OPatch detects the Middleware Home as “/opt/app/oracle/Middleware/WebLogic”

ApplySession applying interim patch ‘12423703’ to OH ‘/opt/app/oracle/Middleware/WebLogic/oms11g’
Execution of ‘sh /opt/app/admin/GCREPO/Grid_patch/12423703/custom/scripts/init -apply 12423703 ‘:

Return Code = 0

Running prerequisite checks…

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch ‘12423703’ for restore. This might take a while…
Backing up files affected by the patch ‘12423703’ for rollback. This might take a while…
Execution of ‘sh /opt/app/admin/GCREPO/Grid_patch/12423703/custom/scripts/pre -apply 12423703 ‘:

Return Code = 0

Patching component oracle.sysman.oms.core, 11.1.0.1.0…
Updating jar file “/opt/app/oracle/Middleware/WebLogic/oms11g/sysman/jlib/emCORE.jar” with /sysman/jlib/emCORE.jar/oracle/sysman/eml/ecm/policy/PolicyViolationsController.class”
Updating jar file “/opt/app/oracle/Middleware/WebLogic/oms11g/sysman/jlib/emCORE.jar” with “/sysman/jlib/emCORE.jar/oracle/sysman/eml/ecm/policy/PolicyViolationsJspBean.class”
Updating jar file “/opt/app/oracle/Middleware/WebLogic/oms11g/sysman/jlib/emCORE.jar” with “/sysman/jlib/emCORE.jar/oracle/sysman/eml/ecm/policy/PolicyViolationsSearch.class”
Updating jar file “/opt/app/oracle/Middleware/WebLogic/oms11g/sysman/jlib/emCORE.jar” with “/sysman/jlib/emCORE.jar/oracle/sysman/emSDK/job/MultiTaskJob.class”
Updating jar file “/opt/app/oracle/Middleware/WebLogic/oms11g/sysman/jlib/emCORE.jar” with “/sysman/jlib/emCORE.jar/oracle/sysman/eml/jobs/JobUtil.class”

::: (It replaces a very long list of files)

Verifying the update…
Inventory check OK: Patch ID 12423703 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 12423703 are present in Oracle Home.
Execution of ‘sh /opt/app/admin/GCREPO/Grid_patch/12423703/custom/scripts/post -apply 12423703 ‘:

Copying file :/opt/app/oracle/Middleware/WebLogic/wlserver_10.3/common/nodemanager/nodemanager.domains to:/opt/app/oracle/Middleware/WebLogic/wlserver_10.3/common/emnodemanager/nodemanager.domains

Return Code = 0

OPatch succeeded.

5. Connect to rcuJDBCEngine as SYSMAN and run the apply.sql script as follows:

$ /opt/app/oracle/Middleware/WebLogic/oms11g/bin/rcuJDBCEngine sysman/@11.11.11.111:1521:GCREPO JDBC_SCRIPT apply.sql $PWD $ORACLE_HOME
###### SQL Patching operation has started. The Pre-requisites check ######
###### may take upto 3 minutes. Please do not cancel the operation. ######
###### Refer to My Oracle Support note 1326515.1 for more information ######

##### Start Patch Pre-requisites Check ######
job_queue_processes value before patching
—————- —————
SID Job Queue value
—————- —————
GCREPO 10
——————————–

Value of job_queue_processes is set to 0
###### Patch Pre-requisites Check SUCCESSFUL ######

###### Execution of the SQL files in the patch was SUCCESSFUL ######

###### Start Patch Post Validation ######
job_queue_processes value after patching
—————- —————
SID Job Queue value
—————- —————
GCREPO 10
——————————–

##### Patch Post Validation SUCCESSFUL ######
###### SQL Patching operation SUCCESSFUL ######

Completed SQL script execution normally.
41 scripts were processed

6. Start OMS

$ emctl start oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Starting WebTier…
WebTier Successfully Started
Starting Oracle Management Server…
Oracle Management Server Successfully Started
Oracle Management Server is Up

Now we are done with OMS patching. It is time to Patch the Agent Home. Patch 9345913 is for Agent home.

Patch Agent with Patch 9345913

1. Perquisites: Make sure opatch,unzip are in path. Also ensure you are using latest OPatch version.
Set ORACLE_HOME to Agent home

2. Ensure that the PSU does not conflict with the already-installed one-off patches. To do so, run the following command to generate a report that lists all conflicting patches.

$ opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./9345913

Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation. All rights reserved.

PREREQ session

Oracle Home : /opt/app/oracle/Middleware/WebLogic/agent11g
Central Inventory : /opt/app/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.2.0.1.1
OUI version : 11.1.0.8.0
OUI location : /opt/app/oracle/Middleware/WebLogic/agent11g/oui
Log file location : /opt/app/oracle/Middleware/WebLogic/agent11g/cfgtoollogs/opatch/opatch2011-08-04_13-41-51PM.log

Patch history file: /opt/app/oracle/Middleware/WebLogic/agent11g/cfgtoollogs/opatch/opatch_history.txt

OPatch detects the Middleware Home as “/opt/app/oracle/Middleware/WebLogic”

Invoking prereq “checkconflictagainstohwithdetail”

Prereq “checkConflictAgainstOHWithDetail” passed.

OPatch succeeded.

3. Stop agent

$ $AGENT_HOME/bin/emctl stop agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Stopping agent … stopped.

4. Apply patch

$ cd 9345913
OMS:htrdbl02:oracle:/opt/app/admin/GCREPO/Grid_patch/9345913
$ opatch apply
Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation. All rights reserved.

Oracle Home : /opt/app/oracle/Middleware/WebLogic/agent11g
Central Inventory : /opt/app/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.2.0.1.1
OUI version : 11.1.0.8.0
OUI location : /opt/app/oracle/Middleware/WebLogic/agent11g/oui
Log file location : /opt/app/oracle/Middleware/WebLogic/agent11g/cfgtoollogs/opatch/opatch2011-08-04_13-44-12PM.log

Patch history file: /opt/app/oracle/Middleware/WebLogic/agent11g/cfgtoollogs/opatch/opatch_history.txt

OPatch detects the Middleware Home as “/opt/app/oracle/Middleware/WebLogic”

ApplySession applying interim patch ‘9345913’ to OH ‘/opt/app/oracle/Middleware/WebLogic/agent11g’

Running prerequisite checks…

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = ‘/opt/app/oracle/Middleware/WebLogic/agent11g’)

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch ‘9345913’ for restore. This might take a while…
Backing up files affected by the patch ‘9345913’ for rollback. This might take a while…
Execution of ‘sh /opt/app/admin/GCREPO/Grid_patch/9345913/custom/scripts/pre -apply 9345913 ‘:

Return Code = 0

::: (Lot files this patch will be changing)

Running make for target libnmevq
Running make for target libnmevc
Running make for target libnmemso
Running make for target emagent
ApplySession adding interim patch ‘9345913’ to inventory

Verifying the update…
Inventory check OK: Patch ID 9345913 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9345913 are present in Oracle Home.
Execution of ‘sh /opt/app/admin/GCREPO/Grid_patch/9345913/custom/scripts/post -apply 9345913 ‘:

Return Code = 0

The local system has been patched and can be restarted.

OPatch succeeded.

5. Start agent

$AGENT_HOME/bin/emctl start agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Starting agent ……… started.

This Patch needs to be applied on each the target hosts.