11gR2

BUG: Creating 11gR2 RAC Using VMware On Toshiba Laptop

I was setting up 11gR2 RAC grid infra on my new Toshiba L745 series laptop on OEL5.4, the installation was going very smoothly but suddenly the VMWare machines crashed while performing remote operations on second node with following message:

<span style="font-family: arial,helvetica,sans-serif;">"vmxaiomgr.retrycontabort" Please disconnect and reconnect if the storage is external to comupter</span>

The next thing was to look at the error file generated by vmware, and it states the following about this error:

<span style="font-family: arial,helvetica,sans-serif;">"Insufficient quota to complete the requested service (1453)" </span>

Though I had enough Diskspace, RAM availability, CPU etc. etc. it was hard to believe that I am running out of allocated resources.
Googling this error message, I saw THIS VMware Communities blog which explains the same error which I faced. It explains about a bug in Toshiba’s WiFi service “cfWiMAXService” i.e “ConfigFree WiMAX” which is causing all this and not allowing to perform the remote copy operation of Grid Infra install.

<span style="font-family: arial,helvetica,sans-serif;">The solution is to disable ConfigFree Service on Toshiba Laptop</span>

After disabling this service, the installation worked fine. Though I am not aware what will be the effect of disabling this serviceon laptop but the laptop is working fine since last two weeks.

Thanks to the above mentioned link which helped me solving this issue.

ORA-00132 while starting 11gR2 database

Short post on a error which can take lot of time to debug. You have upgraded a database and after restoring all files (init,tnsnames) try to startup 11g database and it fails with following error

SQL> startup nomount
ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00132: syntax error or unresolved network name 'scan-clu:1521'

11gR2 requires remote_listener to be set to <scan_name:port>. Problem is that the sqlnet.ora file has been modified/replaced (from earlier version backup) which does not have EZCONNECT and has only TNSNAMES. Ensure that your sqlnet.ora file contain’s EZCONNECT

NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)

Adding a reinstalled/reimaged node back to 11gR2 Cluster

There could be a scenario of node crash due to OS/hardware issues and a reinstall/reimage of the same. In such cases, just a normal node addition would not  help since the OCR still contain the references of original node. We need to remove them first and then perform a node addition.

Have tried to document one such usecase.

Assumptions :
Cluster Hostnames : node1 , node2
VIP : node1-v , node2-v

– Voting disk and OCR are on ASM( ASMLIB is being used to manage the shared disks )
– After the OS reinstall, user equivalence has been set and all required packages have been installed along with setup of ASMLIB
– The crashed node was node2

STEPS
———-
1. Clearing the OCR entries for re-imaged host.

# crsctl delete node -n node2

To verify the success of above step, execute “olsnodes” on surviving node and the reimaged host shouldnot show up in list.

2. Remove the VIP information of reimaged host from OCR

Execute the following on existing node :
	/u01/grid/11.2/bin/srvctl remove vip -i node2-v -f

3. Clear the inventory for reimaged host for GI and DB Homes.

From the surviving node, execute :

/u01/grid/11.2/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/u01/grid/11.2 "CLUSTER_NODES=node1" CRS=TRUE -silent -local

Perform the similar for Database Home as well :

/u01/oracle/product/11.2/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/u01/oracle/product/11.2 CLUSTER_NODES=node1 -silent -local

4. Now starts the actual step of adding node. Run the Run the Cluster Verification Utility

./cluvfy  stage -pre nodeadd -n node2 -verbose

If possible, redirect the output of above to some file so that it can be reviewed and any issues reported can be rectified.

For this case, since the OCR and Voting disk resides on ASM and ASMLIB is in use, the most impacting errors were

ERROR:
PRVF-5449 : Check of Voting Disk location “ORCL:DISK6(ORCL:DISK6)” failed on the following nodes:
node2:No such file or directory

PRVF-5431 : Oracle Cluster Voting Disk configuration check failed

Will explain the impact of this error in the subsequent steps..

5. Run “addNode.sh” from existing node.

[oracle@node1] /u01/grid/11.2/oui/bin% ./addNode.sh -silent "CLUSTER_NEW_NODES={node2}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={node2-v}"
[oracle@node1] /u01/grid/11.2/oui/bin%

In my case, the above command came out without giving any messages. Actually the addNode.sh didnot run at all.

Cause : Since ASMLIB is in use, we had hit the issue discussed in MOS Note : 1267569.1
The error seen in step 4 helped in finding this.

Solution :

Set the following parameters and run addNode.sh again.

IGNORE_PREADDNODE_CHECKS=Y
export IGNORE_PREADDNODE_CHECKS

[oracle@node1] /u01/grid/11.2/oui/bin% ./addNode.sh -silent "CLUSTER_NEW_NODES={node2}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={node2-v}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 12143 MB

Performing tests to see whether nodes node2 are available
............................................................... 100% Done.

Cluster Node Addition Summary
Global Settings
   Source: /u01/grid/11.2
   New Nodes
Space Requirements
   New Nodes
      node2

	Instantiating scripts for add node (Tuesday, December 21, 2010 3:35:16 AM PST)
			.                                                                 1% Done.
			Instantiation of add node scripts complete

			Copying to remote nodes (Tuesday, December 21, 2010 3:35:18 AM PST)
			...............................................................................................                                 96% Done.
			Home copied to new nodes

			Saving inventory on nodes (Tuesday, December 21, 2010 3:37:57 AM PST)
			.                                                               100% Done.
			Save inventory complete
			WARNING:
			The following configuration scripts need to be executed as the "root" user in each cluster node.
			/u01/grid/11.2/root.sh # On nodes node2
			To execute the configuration scripts:
			    1. Open a terminal window
			    2. Log in as "root"
			    3. Run the scripts in each cluster node

			The Cluster Node Addition of /u01/grid/11.2 was successful.
			Please check '/tmp/silentInstall.log' for more details.

6. Run root.sh on reimaged node to start up CRS stack.

This will completed Grid Infrastucture setup on the node.

7. Proceed to run addNode.sh for DB Home( on existing Node)

/u01/oracle/product/11.2/addNode.sh -silent "CLUSTER_NEW_NODES={node2}"

8. Once the DB Home addition is complete, use srvctl to check the status of registered DB and instances and add them if required.

October 2010 Blogroll Report

It’s been time since we saw log-buffer edition with last being published on 1st October 2010 and  Coskan’s weekly blogroll report

I found them really useful to read top blog articles for week. So I decided to tag few articles posted in October 2010 which I  had read and found useful. I am subscribed to orana.info so most of articles are already aggregated there (barring few). Unlike Pythian’s log buffer, I will be concentrating only on Oracle Database Related posts. Lets start

11.2.0.2 patchset introduced lot of new features in Grid Infrastructure. Martin Bach discusses Redundant Interconnect feature introduced in 11.2.0.2 in his following post

Then there is series on setting up Stretched RAC (with 4 parts published till now) from him

Julian Dyke has started his blog and have posted quite a number of articles related to 11.2.0.2
He discussed issue related to Multicasting (required to enable HAIP interconnect feature) in below post

Julian also published test program to check if multicasting is enabled on your box

Ritesh has posted a handy script to map ASM devices on EMC along with Disk number along with a  tweak in max sector size configuration on RHEL5 which can boost ASM Performance.Read more below


Doug writes about a issue related to SQL*Net events and methodology used by him to debug ‘SQL*Net more data from client’ wait event

tkprof has undergone a suprising change in 11.2.0.2. Read Christian’s article about tkprof changes in 11.2.0.2

Another posting from Christian highlights changes related to Deferred Segment creation introduced in 11.2.0.2 in following post


Jarneil writes post about SQL Plan Management in 11g and discusses a case when adding index to table is not picked automatically by CBO

How do you answer question “What is difference between Nested Loop and Hash Joins“. Tanel Poder says “Hash joins can not look up rows from the inner (probed) row source based on values retrieved from the outer (driving) row source, nested loops can.”
Read on below article (make sure to check comments section to know what Jonathan Lewis and Christo kutrovsky had to say)


Check out article on Reading parallel execution with Bloom pruning by Greg Rahn


In 11.2 you can query bind values of running query instead of dumping using errorstack. Tanel Poder gives a example describing with example


Dominic talks about NLS (I know many of you would be scared but believe me you can’t avoid it). Read about using NLS_LENGTH_SEMANTICS

Kellyn troubleshoots ora-600 on RMAN and is able to resolve the bug without applying patch.

Meanwhile she is also trying out setting up wordpress blog on dbakevlar.com (which is so much better than blogspot)

Craig Shallahamer explains correlation between database commit time and log file sync time

Do you have export scripts which you are scared to migrate to datapump. Probably Legacy mode of datapump (11gR2 feature) can help you. Aman Sharma explains the usage in below post
Happy Reading!

Steps to Upgrade 11.2.0.1 RAC to 11.2.0.2

This post is in continuation to Grid Infrastructure upgrade post written earlier. Ensure that you have downloaded the software and  have upgraded the infrastructure database.

Following files contain database software for 11.2.0.2.

p10098816_112020_Linux-x86-64_1of7.zip

p10098816_112020_Linux-x86-64_2of7.zip

Please note that this is full release , so if you are installing new software, you can directly install 11.2.0.2 without need of first installing 11.2.0.1 database.Unzip the software and start the runInstaller from the database directory.

cd $SW_HOME/database

./runInstaller

We will be presented with OUI screen asking for MOS credentials. Please note that we would be installing software into new ORACLE_HOME, which is a new feature called out of place upgrade.

Next screen ask’s again for MOS credentials. This can be used to check if there are any patches which need’s to be applied before software can be upgraded. Since we have already applied the PSU2, we will choose skip Software updates

We are now presented with 3 options

a)Create and Configure database – installs software and creates a database using dbca

b)Install database software only – Only installs 11.2.0.2 software

c) Upgrade exisiting database – Installs software and launches dbua to upgrade database

We chose option b) i.e Install database software only

Next screen presents you with 3 options

a)Single Instance database installation

b)Oracle RAC Installation

c)Oracle RAC One Node database Installation

I will be discussing RAC one node in a future post. For our installation we are upgrading RAC database and require RAC software

Next screen gives option to choose between Enterprise edition and Standard Edition

Next screen asks for database software installation directory. Unlike Grid infrastructure, where it is mandatory to install in new ORACLE_HOME, RAC database software can be installed in existing home. But we will be choosing Out of place upgrade i.e install in new home.

Choose the OSDBA and OSOPER group

Next screen checks the pre-requisites. You can ask oracle to create fixup script. Refer to my earlier post for detail

Finally we are prompted to run root.sh from both nodes

After you have run root.sh from both nodes, you can then use dbua to upgrade the database. Note that we have not yet brought down the database. This is great benefit of using out of place upgrade . Second benefit is that you are not touching the existing binaries,so you are not required to take backup of binaries and can can easily rollback the changes (if required). You can also continue using the old binaries for databases which you cannot get downtime and can upgrade them later.

Set the ORACLE_HOME and PATH to include 11.2.0.2 software location and start dbua. Take backup of database before you start upgrade process using dbua

$dbua

We are presented with welcome screen. Press next

DBUA presents you with list of databases currently registered in /etc/oratab. We select db11g which we need to upgrade

Next screen asks for following options

a) Option to recompile invalid objects and degree of parallelism for running utlrp.sql. We keep default value of 3

b) Option to disable archiving during upgrade

c)Upgrade the timezone file. We have not selected it now and will do upgrade manually

Next screen presents with option to select Fast Recovery area and size

Next we will be presented with summary screen and asked to take backup of database. If you have not taken it till now, its good time to take backup. Also now database will be stopped and will be started from new oracle home.Please ensure that you have appropriate setting for JAVA_POOL_SIZE and SHARED_POOL_SIZE during upgrade or can use SGA_TARGET/MEMORY_TARGET to avoid ora-4031

Following screen shows database upgrade progress screen.

Once upgrade completes, it gives summary of upgrade process. As you can see it has warned for DST upgrade.

Oracle 11.2.0.2 contains version 14 file. Starting 11g you can have multiple database running out of single oracle home to have different timezone version files.

You can refer to Updating the RDBMS DST version in 11gR2 (11.2.0.1 and up) using DBMS_DST [ID 977512.1]

Giving summary of steps taken by us to upgrade from version 11 to version 14

---Check the current version of file ---
SQL> select version from V$TIMEZONE_FILE;

   VERSION
----------
	11

SQL> SELECT PROPERTY_NAME, SUBSTR(property_value, 1, 30) value
FROM DATABASE_PROPERTIES
WHERE PROPERTY_NAME LIKE 'DST_%'
ORDER BY PROPERTY_NAME;  2    3    4  

PROPERTY_NAME		       VALUE
------------------------------ ------------------------------
DST_PRIMARY_TT_VERSION	       11
DST_SECONDARY_TT_VERSION       0
DST_UPGRADE_STATE	       NONE

---Confirm if $ORACLE_HOME/oracore/zoneinfo contains timezlrg_14.dat i.e version 14 file

--Prepare for upgrade
SQL> exec DBMS_DST.BEGIN_PREPARE(14);

PL/SQL procedure successfully completed.

--Confirm status

SQL> SELECT PROPERTY_NAME, SUBSTR(property_value, 1, 30) value
FROM DATABASE_PROPERTIES
WHERE PROPERTY_NAME LIKE 'DST_%'
ORDER BY PROPERTY_NAME;  2    3    4  

PROPERTY_NAME		       VALUE
------------------------------ ------------------------------
DST_PRIMARY_TT_VERSION	       11
DST_SECONDARY_TT_VERSION       14
DST_UPGRADE_STATE	       PREPARE

---Re-running prepare statement will give below error.
SQL> exec DBMS_DST.BEGIN_PREPARE(14);
BEGIN DBMS_DST.BEGIN_PREPARE(14); END;

*
ERROR at line 1:
ORA-56920: a prepare or upgrade window or an on-demand or datapump-job loading of a secondary time zone data file is in
an active state
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 79
ORA-06512: at "SYS.DBMS_DST", line 1340
ORA-06512: at line 1

-- truncate logging tables if they exist.

TRUNCATE TABLE SYS.DST$TRIGGER_TABLE;
TRUNCATE TABLE sys.dst$affected_tables;
TRUNCATE TABLE sys.dst$error_table;

-- log affected data

BEGIN
DBMS_DST.FIND_AFFECTED_TABLES
(affected_tables => 'sys.dst$affected_tables',
log_errors => TRUE,
log_errors_table => 'sys.dst$error_table');
END;
/
---Query table to check if any affected table
SQL> SQL> SELECT * FROM sys.dst$affected_tables;

no rows selected

SQL>
SQL> SELECT * FROM sys.dst$error_table;

no rows selected

-- End the prepare phase
SQL> EXEC DBMS_DST.END_PREPARE;
A prepare window has been successfully ended.

--Startup the database in upgrade mode. Ensure you have set cluster_database=false

SQL> EXEC DBMS_DST.BEGIN_UPGRADE(14);
An upgrade window has been successfully started.

PL/SQL procedure successfully completed.

-- To confirm whether it has been upgraded to version 14
SQL> SELECT PROPERTY_NAME, SUBSTR(property_value, 1, 30) value
FROM DATABASE_PROPERTIES
WHERE PROPERTY_NAME LIKE 'DST_%'
ORDER BY PROPERTY_NAME;  2    3    4  

PROPERTY_NAME		       VALUE
------------------------------ ------------------------------
DST_PRIMARY_TT_VERSION	       14
DST_SECONDARY_TT_VERSION       11
DST_UPGRADE_STATE	       UPGRADE

--Restart the database after removing cluster_database

This completes the database upgrade process. I would recommend dbua to upgrade the database as  it takes care of copying the init.ora files,password file and also modified the OCR to point to new oracle home and upgraded srvctl version. Oracle Upgrade guide does not list any method documenting steps to be taken for srvctl. I referred following link i.e Oracle Database Upgrade documentation for the upgrade purpose

http://download.oracle.com/docs/cd/E11882_01/server.112/e17222/upgrade.htm#i1011981

In case you use manual method and use srvctl modify database to point to new oracle home, you will get errors like below

srvctl stop database -d test11g
PRCD-1027 : Failed to retrieve database test11g
PRCD-1229 : An attempt to access configuration of database test11g was rejected because its version 11.2.0.1.0 differs from the program version 11.2.0.2.0. Instead run the program from /oracle/product/app/11.2.0/dbhome_1.

You can use following command to upgrade the srvctl version

srvctl upgrade database -d test11g -o /oracle/product/app/11.2.0.2/dbhome_1

Happy Upgrading.

Upgrading Grid Infrastructure 11.2.0.1 to 11.2.0.2 on RHEL4

11.2.0.2 Patchset was released few days back. I decided to upgrade a test RAC database to 11.2.0.2 yesterday and was able to do it successfully. I will be documenting the steps here for easy reference

Environment Details

2 node RAC on Red Hat Enterprise Linux AS release 4 (Nahant Update 8), 64 bit

To upgrade existing 11.2.0.1 Oracle Grid Infrastructure installations to Oracle Grid Infrastructure 11.2.0.2, you must first do at least one of the following:

– Patch the release 11.2.0.1 Oracle Grid Infrastructure home with the fix for bug 9413827.
– Install Oracle Grid Infrastructure Patch Set 1 (GIPS1) or Oracle Grid Infrastructure Patch Set 2 (GIPS2).

I will be using Oracle Grid Infrastructure Patch set 2.

Software and Patches

1) Download Patch 9655006 – 11.2.0.1.2 for Grid Infrastructure (GI) Patch Set Update from MOS

2) Latest Opatch for 11.2 – 6880880 Universal Installer: Patch OPatch 11.2

You can refer to How To Download And Install OPatch [ID 274526.1]

3)11.2.0.2 Patchset files. You can find method to download them directly to server here

p10098816_112020_Linux-x86-64_1of7.zip  – 11.2.0.2 Database Installation Files

p10098816_112020_Linux-x86-64_2of7.zip – 11.2.0.2 Database Installation Files

p10098816_112020_Linux-x86-64_3of7.zip – 11.2.0.2 Grid Installation Files

Lets get started.

Install Latest Opatch

Unzip the zip file and copy OPatch folder to $ORACLE_HOME by renaming the earlier OPatch directory. You can refer to How To Download And Install OPatch [ID 274526.1]

Apply Grid Infrastructure (GI) Patch Set Update –  9655006

It applies patch 9654983 and 9655006 to both Database and Grid home. We proceed by checking for any patch conflicts
[oracle@oradbdev01]~/software/11.2.0.2/psu% opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./9655006
Invoking OPatch 11.2.0.1.3

Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /oracle/product/app/11.2.0/dbhome_1
Central Inventory : /oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /oracle/product/app/11.2.0/dbhome_1/oui
Log file location : /oracle/product/app/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2010-10-02_06-03-34AM.log

Patch history file: /oracle/product/app/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch_history.txt

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.<span style="font-family: Georgia, 'Times New Roman', 'Bitstream Charter', Times, serif; font-size: small;"><span style="line-height: 19px; white-space: normal;">
</span></span>

We need to use opatch auto to patch both Grid Infrastructure and RAC database home.It can be used to patch separately but I am going with patching both. This is great improvement over 10g CRS patch bundles as it had lot of steps to be run as root or oracle software owner. ( I had myself messed up once by running a command as root instead of oracle 🙁 ) Opatch starts with patching the database home and then patches the grid infrastructure home. You can stop the database instance running out of oracle home.(opatch asks for shutting down database, so in case you do not stop,don’t worry it will warn)

Note: In case you are on RHEL5/OEL5 and using ACFS for database volumes, it is recommended to use opatch auto <loc> -oh <grid home> to first patch the Grid home instead of patching them together

Unzip patch 965506 to directory say /oracle/software/gips2.It will create two directories for patch 9654983 and 9655006. Connect as root user and set the oracle home and Opatch directory . Stop the database instance running on the node

srvctl stop instance -d db11g -i db11g1

Next run opatch auto command as root

#opatch auto /oracle/software/gips2
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /oracle/product/grid
Central Inventory : /oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /oracle/product/grid/oui
Log file location : /oracle/product/grid/cfgtoollogs/opatch/opatch2010-10-02_06-35-03AM.log

Patch history file: /oracle/product/grid/cfgtoollogs/opatch/opatch_history.txt

Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9654983  

Do you want to proceed? [y|n]
y
User Responded with: Y

Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y

You selected -local option, hence OPatch will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/oracle/product/grid')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...

I have not copied the whole output. It asks you to specify MOS credentials for setting up Oracle Configuration Manager. You can skip it as mentioned above. You will find following errors in the end

OPatch succeeded.
ADVM/ACFS is not supported on Redhat 4

Failure at scls_process_spawn with code 1
Internal Error Information:
  Category: -1
 Operation: fail
  Location: canexec2
  Other: no exe permission, file [/oracle/product/grid/bin/ohasd]
  System Dependent Information: 0

CRS-4000: Command Start failed, or completed with errors.

Timed out waiting for the CRS stack to start.<span style="font-family: Georgia, 'Times New Roman', 'Bitstream Charter', Times, serif; font-size: small;"><span style="line-height: 19px; white-space: normal;">
</span></span>

This is a known issue and discussed in 11.2.0.X Grid Infrastructure PSU Known Issues [ID 1082394.1]

To solve this issue, connect as root user and execute following command from your Grid Infrastructure home

cd $GRID_HOME

# ./crs/install/rootcrs.pl -patch
2010-10-02 06:59:54: Parsing the host name
2010-10-02 06:59:54: Checking for super user privileges
2010-10-02 06:59:54: User has super user privileges
Using configuration parameter file: crs/install/crsconfig_params
ADVM/ACFS is not supported on Redhat 4

CRS-4123: Oracle High Availability Services has been started.

Start oracle database instance on the node which has been patched

srvctl start instance -d db11g -i db11g1

Repeat the above steps for node 2.It took me 40 minutes for each node to complete opatch auto activity.Once it is done,execute following from one node for all databases to complete patch installation

cd $ORACLE_HOME/rdbms/admin

sqlplus /nolog

SQL> CONNECT / AS SYSDBA

SQL> @catbundle.sql psu apply

SQL> QUIT

This completes patching of Grid Infrastructure Patch or PSU2.

Upgrading Grid Infrastructure home to 11.2.0.2

Ensure following environment variables are not set :ORA_CRS_HOME; ORACLE_HOME; ORA_NLS10; TNS_ADMIN

# echo $ORA_CRS_HOME; echo $ORACLE_HOME; echo $ORA_NLS10; echo $TNS_ADMIN

Strating with 11.2.0.2, Grid infrastructure (Clusterware and ASM Home) upgrade is out of place upgrade i.e we install in new ORACLE_HOME. Unlike database home we cannot perform an in-place upgrade of Oracle Clusterware and Oracle ASM to existing homes.

Unset following variables too

$ unset ORACLE_BASE

$ unset ORACLE_HOME

$ unset ORACLE_SID

Relax permissions for GRID_HOME

#chmod -R 755 /oracle/product/grid

#chown -R oracle /oracle/product/grid

#chown oracle /oracle/product

Start the runInstaller from the 11.2.0.2 grid software directory.

On first screen, it will ask for MOS details. I have chosen skip Software update

Since we are upgrading existing installation, choose “Upgrade Oracle Grid Infrastructure or Oracle Automatic Storage Management”. This will install software and also configure ASM

Next screen displays nodes which OUI will patch

Select the OSDBA,OSASM and OSOPER group. I have chosen dba

Since I have chosen all 3 groups to be same, it gives warning. Select Yes to continue

Specify the grid software installation directory. I have used /oracle/product/oragrid/11.2.0.2 (Better to input release as we will be having out of place upgrades for future patchsets too)

OUI reports some issues with swap size ,shmmax and NTP. You can fix them or choose to ignore. OUI can create a fixup script for you.

Installation starts copying file. After files have been copied to both nodes, it asks for running the rootupgrade.sh script from all nodes

Stop the database instance running from the node at this time and then run script as root. Please note that ASM and clusterware should not be stopped as rootupgrade.sh requires them to be up and takes care of shutting down and starting with new home.In my case, running rootupgrade.sh successfully succeeded on node 1 but it hung on node 2 . I did a cancel and re-ran it. Pasting the contents from node 2 second run

[root@oradbdev02 logs]# /oracle/product/oragrid/11.2.0.2/rootupgrade.sh
Running Oracle 11g root script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /oracle/product/oragrid/11.2.0.2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/product/oragrid/11.2.0.2/crs/install/crsconfig_params
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-1115: Oracle Clusterware has already been upgraded.

ASM upgrade has finished on last node.

Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

On clicking next in OUI, it reported that cluvfy has failed and some of components are not installed properly. To verify I ran cluvfy manually. Pasting content for which it failed

./cluvfy stage -post crsinst -n oradbdev01,oradbdev02 -verbose 

<span style="font-family: Georgia, 'Times New Roman', 'Bitstream Charter', Times, serif; line-height: 19px; white-space: normal; font-size: 13px;">
<pre style="font: normal normal normal 12px/18px Consolas, Monaco, 'Courier New', Courier, monospace;">-------------------------------truncated output -------------------

ASM Running check passed. ASM is running on all specified nodes

Checking OCR config file "/etc/oracle/ocr.loc"...

ERROR:
PRVF-4175 : OCR config file "/etc/oracle/ocr.loc" check failed on the following nodes:

	oradbdev02:Group of file "/etc/oracle/ocr.loc" did not match the expected value. [Expected = "oinstall" ; Found = "dba"]

	oradbdev01:Group of file "/etc/oracle/ocr.loc" did not match the expected value. [Expected = "oinstall" ; Found = "dba"]

Disk group for ocr location "+DG_DATA01" available on all the nodes

OCR integrity check failed
-------------------------------truncated output -------------------

Checking OLR config file...

ERROR: 

PRVF-4184 : OLR config file check failed on the following nodes:

	oradbdev02:Group of file "/etc/oracle/olr.loc" did not match the expected value. [Expected = "oinstall" ; Found = "dba"]

	oradbdev01:Group of file "/etc/oracle/olr.loc" did not match the expected value. [Expected = "oinstall" ; Found = "dba"]

Checking OLR file attributes...

ERROR: 

PRVF-4187 : OLR file check failed on the following nodes:

	oradbdev02:Group of file "/oracle/product/oragrid/11.2.0.2/cdata/oradbdev02.olr" did not match the expected value. [Expected = "oinstall" ; Found = "dba"]

	oradbdev01:Group of file "/oracle/product/oragrid/11.2.0.2/cdata/oradbdev01.olr" did not match the expected value. [Expected = "oinstall" ; Found = "dba"]

OLR integrity check failed

-------------------------------truncated output -------------------
Checking NTP daemon command line for slewing option "-x"
Check: NTP daemon command line
  Node Name                             Slewing Option Set?
  ------------------------------------  ------------------------
  oradbdev02                            no
  oradbdev01                            no
Result:
NTP daemon slewing option check failed on some nodes
PRVF-5436 : The NTP daemon running on one or more nodes lacks the slewing option "-x"
Result: Clock synchronization check using Network Time Protocol(NTP) failed

PRVF-9652 : Cluster Time Synchronization Services check failed

We see that cluvfy is reporting error that it expected oinstall group but found dba group. I had not specified install group during installation, so can ignore it.For NTP you can correct it by setting following in ntpd.conf and restart the ntpd daemon

grep OPTIONS /etc/sysconfig/ntpd
OPTIONS=”-u ntp:ntp -p /var/run/ntpd.pid -x”

Refer How to Configure NTP to Resolve CLUVFY Error PRVF-5436 PRVF-9652 [ID 1056693.1]

At this moment Grid Infrastructure has been successfully upgraded to 11.2.0.2. Next we will upgrade the RAC database home. You can refer to Steps to Upgrade 11.2.0.1 RAC to 11.2.0.2