RAC

Continuing my experiments with our 2 Node 10g RAC Test system, I carried out upgrade of Oracle Clusterware and Oracle RAC Database from 10.2.0.1 to 10.2.0.4. I have tried to document the steps for upgrading Oracle Clusterware(Rolling Upgrade) and RAC Database upgrade in this post. In case you observe any mistakes, please let me know

First step is to download the 10.2.0.4 Patchset from metalink. In our case ,we downloaded Patch 6810189 (10g Release 2 (10.2.0.4) Patch Set 3 for Linux x86). You can follow Patch Readme for detailed steps.

We will be doing Rolling upgrade for Oracle Clusterware i.e we will only bring one node down for patching while other node will be available and accepting database connections. Before you start the process, take backup of following so as to restore it in case of failed upgrade

a) Full OS backup (as some binaries are present in /etc ,etc)

b) Full Database Backup (Cold or hot backup)

c) Backup of OCR and voting disk

Let’s begin it

1)Shutdown the DBconsole and Isqlplus

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">$ emctl stop dbconsole
$ isqlplusctl stop
</span>

2) Shutdown the associated service on the node

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">[oracle@blrraclnx1 ~]$ srvctl stop service -d orcl -s orcl_taf -i orcl1</span>

3) Shutdown Database Instance and ASM instance on node (if present)

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">[oracle@blrraclnx1 ~]$  srvctl stop instance -d orcl -i orcl1
</span>

To stop ASM, use following command

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">[oracle@blrraclnx1 ~]$ srvctl stop asm -n blrraclnx1
</span>

4)Next step is to stop Nodeapps services on the node

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">[oracle@blrraclnx1 ~]$ srvctl stop nodeapps -n blrraclnx1</span>

Before proceeding to installing Oracle Clusterware Patch, let’s confirm if services have been stopped

HA Resource                                   Target     State
-----------                                   ------     -----
<strong>ora.blrraclnx1.ASM1.asm                       OFFLINE    OFFLINE
ora.blrraclnx1.LISTENER1_BLRRACLNX1.lsnr      OFFLINE    OFFLINE
ora.blrraclnx1.gsd                            OFFLINE    OFFLINE
ora.blrraclnx1.ons                            OFFLINE    OFFLINE
ora.blrraclnx1.vip                            OFFLINE    OFFLINE</strong>
ora.blrraclnx2.ASM2.asm                       ONLINE     ONLINE on blrraclnx2
ora.blrraclnx2.LISTENER1_BLRRACLNX2.lsnr      ONLINE     ONLINE on blrraclnx2
ora.blrraclnx2.gsd                            ONLINE     ONLINE on blrraclnx2
ora.blrraclnx2.ons                            ONLINE     ONLINE on blrraclnx2
ora.blrraclnx2.vip                            ONLINE     ONLINE on blrraclnx2
ora.orcl.db                                   ONLINE     ONLINE on blrraclnx2
<strong>ora.orcl.orcl1.inst                           OFFLINE    OFFLINE</strong>
ora.orcl.orcl2.inst                           ONLINE     ONLINE on blrraclnx2
ora.orcl.orcl_taf.cs                          ONLINE     ONLINE on blrraclnx2
<strong>ora.orcl.orcl_taf.orcl1.srv                   OFFLINE    OFFLINE</strong>
ora.orcl.orcl_taf.orcl2.srv                   ONLINE     ONLINE on blrraclnx2

5)Set DISPLAY variable and execute runinstaller from Patch Directory

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">[oracle@blrraclnx1 Disk1]$ ./runInstaller
</span>

This will open OUI screen. Select Oracle Clusterware Home for Patching. Find below screenshot for same

This will automatically select all the nodes available in cluster and propogate patch binaries to the other node.

6) On the Summary screen, click Install.OUI will prompt you now to run, following two scripts as Root which will upgrade Oracle Clusterware

<span style="font-size: small; font-family: arial,helvetica,sans-serif;"># $ORA_CRS_home/bin/crsctl stop crs
# $ORA_CRS_home/install/root102.sh
</span>

Now we need to repeat the steps 1-4 and step 6 on Node 2. Step 5 is not required as binaries have been already copied over to node 2.

RAC Database Patching cannot be done in a rolling fashion and requires Database to be shutdown.

1)Shutdown the DBconsole and Isqlplus

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">$ emctl stop dbconsole
$ isqlplusctl stop

</span>

2) Shutdown the associated service for database

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">[oracle@blrraclnx1 ~]$ srvctl stop service -d orcl </span>

3) Shutdown Database Instance and ASM instance on node (if present)

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">[oracle@blrraclnx1 ~]$  srvctl stop database -d orcl
</span>

To stop ASM, use following command on both the nodes

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">[oracle@blrraclnx1 ~]$ srvctl stop asm -n blrraclnx1
&lt;/span&gt;<span style=\"font-size: small; font-family: arial,helvetica,sans-serif;\">[oracle@blrraclnx1 ~]$ srvctl stop asm -n blrraclnx2</span>

4)Next step is to stop Listener on both the nodes

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">[oracle@blrraclnx1 ~]$ srvctl stop listener -n blrraclnx1 -l LISTENER1_BLRRACLNX1
</span><span style="font-size: small; font-family: arial,helvetica,sans-serif;">[oracle@blrraclnx1 ~]$ srvctl stop </span><span style="font-size: small; font-family: arial,helvetica,sans-serif;">listener</span><span style="font-size: small; font-family: arial,helvetica,sans-serif;"> -n blrraclnx2 -l LISTENER1_BLRRACLNX2
</span>

5)Set DISPLAY variable and execute runinstaller from Patch Directory

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">[oracle@blrraclnx1 Disk1]$ ./runInstaller
</span>

This will open OUI screen. Select Database Home for Patching.

6) On the Summary screen, click Install.When prompted, run the $ORACLE_HOME/root.sh script as the root
user on both the nodes. On completion of this , we need to perform post installation steps.

7)Start listener and ASM Instance on both the nodes

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">[oracle@blrraclnx1 ~]$ srvctl start listener -n blrraclnx1 -l LISTENER1_BLRRACLNX1
[oracle@blrraclnx1 ~]$ srvctl start listener -n blrraclnx2 -l LISTENER1_BLRRACLNX2
[oracle@blrraclnx1 ~]$ srvctl start asm -n blrraclnx1
[oracle@blrraclnx1 ~]$ srvctl start asm -n blrraclnx2</span>

8)For Oracle RAC Installation, we need to set CLUSTER_DATABASE=FALSE before upgrading

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">[oracle@blrraclnx1 ~]sqlplus "/ as sysdba"
SQL&gt;startup nomount
SQL&gt; alter system set cluster_database=false scope=spfile;

System altered.
SQL&gt;shutdown immediate;
SQL&gt;startup upgrade
SQL&gt;spool 10204patch.log
SQL&gt;@?/rdbms/admin/catupgrd.sql
SQL&gt;spool off</span>

Log file needs to be reviewed for any errors. catupgrd.sql took 42 minutes on my system. In case CLUSTER_DATABASE parameter is not set to False, you will get following error while starting database in upgrade mode

ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE

We need to Restart the database now and run utlrp.sql.

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">SQL&gt; SHUTDOWN IMMEDIATE
SQL&gt; STARTUP
SQL&gt; @?/rdbms/admin/utlrp.sql</span>

Confirm whether Database has been upgraded successfully by querying DBA_REGISTRY;

select comp_name,version,status from dba_registry;

Now set Cluster_database parameter to TRUE and start Database

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">SQL&gt;alter system set cluster_database=true scope=spfile;
SQL&gt;Shutdown immediate;
[oracle@blrraclnx1 ~]$ srvctl start database -d orcl
[oracle@blrraclnx1 ~]$ srvctl start service -d orcl</span>

To upgrade DBConsole, run following command

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">emca -upgrade db -cluster
</span>

This completes the upgrade process.

Last week we were trying to setup a 2 Node 10g RAC System on Linux with openfiler used for shared storage. We were using the article written by Jeffery Hunter. This was not the first time I was doing it, but by mistake we chose the machine with 500Mb memory to be used for one of the RAC system and used a 1Gb memory machine for Openfiler. We carried on with the installation though cluvfy and runInstaller gave us warnings regarding the same.

But once the installation completed, I found the database was shutting down frequently with “PMON Failed to acquire Latch”. I tried to debug it, but was not able to figure out anything from Systemstate dump which was generated.

Anyways we decided to rebuild the system.So we decided to cleanup the Machine 2 and meanwhile re-installed the openfiler. I was not having the OEL Cd’s on that day,so couldn’t build the Machine 1. So I went ahead with cleaning Machine 2 for re-installing software. I saw an opportunity of setting up a single node RAC and then adding another node. I followed below steps for cleaning the RAC installation

1)Stop the Nodeapps and Clusterware

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">srvctl stop nodeapss -n</span>

This will shutdown the Database,ASM instance and also the nodeapps. After this , you can stop the clusterware.

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">#crsctl stop crs</span>

Please note that you can directly stop the clusterware (while cleaning up) as this will automatically stop the dependent resources.

2) Remove the installation files and other related files
I had installed CRS in /u01/app/crs and Database home was located in /u01/app/oracle. So I removed both the directories.

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">rm -rf /u01/app/crs
rm -rf /u01/app/oracle</span>

Note that if you are having multiple oracle database installation, then ensure that you do not remove orainventory directoy or any other ORACLE_HOME. In my case this was the only installation. Remove following files related to clusterware

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">        rm /etc/oracle/*
	rm -f /etc/init.d/init.cssd
	rm -f /etc/init.d/init.crs
	rm -f /etc/init.d/init.crsd
	rm -f /etc/init.d/init.evmd
	rm -f /etc/rc2.d/K96init.crs
	rm -f /etc/rc2.d/S96init.crs
	rm -f /etc/rc3.d/K96init.crs
	rm -f /etc/rc3.d/S96init.crs
	rm -f /etc/rc5.d/K96init.crs
	rm -f /etc/rc5.d/S96init.crs
        rm -Rf /etc/oracle/scls_scr
	rm -f /etc/inittab.crs
	cp /etc/inittab.orig /etc/inittab
        rm -f /var/tmp/.oracle</span>

Also remove the OCR and Voting disk files. In my case it was stored in OCFS2 filesystem /u02/oradata/orcl. In case it is on raw devices , you can remove it using dd command. Remove ocr.loc file present in /etc/oracle

You can also refer to Note:239998.1 – 10g RAC: How to Clean Up After a Failed CRS Install

In our case as we were re-installing after successful installation. So we even had to clean the ASM disks. They can be again cleaned up by formatting the header with dd command.

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">dd if=/dev/zero of=/dev/sdb bs=1024 count=100</span>

As we were removing the other node and had to reconfigure SSH, I removed /home/oracle/.ssh directory. I didn’t reconfigure SSH again thinking that as it will not be required for single node install. I restarted the Clusterware install and encountered following error

“The Specified nodes are not clusterable”

In another window, one more error was reported, which actually made it clear where the problem was

“Failed to check remote command execution setup for node <nodename> shells /us/bin/ssh and /usr/bin/rsh”

Screenshot for the error can be seen below

Above error clearly states that error was due to unavailability of ssh or rsh. After this I did setup for ssh for single node and tested this too to avoid any further errors.

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">$ ssh blrraclnx2 date
Sun Aug 10 14:32:29 EDT 2008</span>

Anyways all these errors could have been avoided, had I used cluvfy utility as below

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">$./runcluvfy.sh stage -pre crsinst -n blrraclnx2 -verbose</span>

Learning:- Always use Cluvfy utility to ensure all pre-requisites are met before installing RAC components