opatch

11gR2:Oracle Online Patching

Online patching is new feature introduced in 11.1.0.6 and oracle started delivering patches with 11.2.0.2.

This patch allows you to apply patch to live database i.e we do not need to shutdown database. This feature is only available for RDBMS home and can be installed/enabled using opatch.

RDBMS Online Patching Aka Hot Patching [ID 761111.1] explains internals of how this patching works.

I tested it while working on Database corruption issue caused by Bug 10205230 ORA-600 / corruption possible during shutdown in RAC. This bug is present in 11.2.0.2 database which causes corruption when you try to use shutdown immediate/normal/transactional option to shutdown RAC database. As per MOS note 1318986.1, Data corruption occurs around shutdown one or more of the RAC instances

* One of the following ORA-600 asserts:
– ORA-600 [kclchkblk_3]
– ORA-600 [kclwcrs_6]
– ORA-600 [ktubko_1]
– ORA-600 [kcratr_scan_lostwrt]
– ORA-600[3020] on the standby database

Workaround is to perform local check point and do shutdown abort. Patch 10205230 was available for our platform and I downloaded using below command directly to server

read mos_user_id
read -s h_passwd
wget --http-user="$mos_user_id" --http-password="$h_passwd" "https://updates.oracle.com/Orion/Services/download/p10205230_112020_Linux-x86-64.zip?aru=13671866&patch_file=p10205230_112020_Linux-x86-64.zip" -O p10205230_112020_Linux-x86-64.zip

mos_user_id – Metalink aka Mos user id (email id)
h_passwd – Password

To query if patch is online, go to patch directory and run following

 

$opatch query -all online
Invoking OPatch 11.2.0.1.6
Oracle Interim Patch Installer version 11.2.0.1.6
Copyright (c) 2011, Oracle Corporation. All rights reserved.

Oracle Home : /home/oracle/product/11.2
Central Inventory : /home/oracle/oraInventory
 from : /etc/oraInst.loc
OPatch version : 11.2.0.1.6
OUI version : 11.2.0.2.0
Log file location : /home/oracle/product/11.2/cfgtoollogs/opatch/opatch2012-06-20_16-43-21PM.log
--------------------------------------------------------------------------------
 Patch created on 2 Apr 2011, 12:57:14 hrs PST8PDT
 Need to shutdown Oracle instances: false
 Patch is roll-backable: true
 Patch is a "Patchset Update": false
 Patch is a rolling patch: true
 Patch has sql related actions: false
 Patch is an online patch: true
 Patch is a portal patch: false
 Patch is an "auto-enabled" patch: false
List of platforms supported:
 226: Linux x86-64
List of bugs to be fixed:
 10205230: TB_X64: HIT ORA-00600: [KCLWCRS_6]
This patch is a "singleton" patch.
This patch belongs to the "db" product family
List of executables affected:
 ORACLE_HOME/bin/oracle
List of optional components:
 oracle.rdbms: 11.2.0.2.0
List of optional actions:
 Patch the Database instances with Online Patch hpatch/bug10205230.pch
Possible XML representation of the patch:
<ONEOFF REF_ID="10205230" ROLLBACK="T" XML_INV_LOC="oneoffs/10205230/" ACT_INST_VER="11.2.0.2.0" INSTALL_TIME="2012.Jun.20 16:43:21 PDT">
 <DESC></DESC>
 <REF_LIST>
 <REF NAME="oracle.rdbms" VER="11.2.0.2.0" HOME_IDX="0"/>
 </REF_LIST>
 <BUG_LIST>
 <BUG>10205230</BUG>
 </BUG_LIST>
 <FILE_LIST/>
</ONEOFF>
--------------------------------------------------------------------------------
OPatch succeeded.

“Patch is an online patch: true” indicates patch can be applied to live db.

To apply the patch, we can use following syntax

Non – RAC => opatch apply online -connectString <SID>:<USERNAME>:<PASSWORD>:<NODE>
RAC => opatch apply online -connectString <SID>:<USERNAME>:<PASSWORD>:<NODE1>,<SID2>:<USERNAME>:<PASSWORD>:<NODE2>,…

Since we are patching RAC database, we will use second syntax

$ opatch apply online -connectString orcl01d1:sys:oracle123:prod03,orcl01d2:sys:oracle123:prod04
Invoking OPatch 11.2.0.1.6
Oracle Interim Patch Installer version 11.2.0.1.6
Copyright (c) 2011, Oracle Corporation. All rights reserved.

Oracle Home : /home/oracle/product/11.2
Central Inventory : /home/oracle/oraInventory
 from : /etc/oraInst.loc
OPatch version : 11.2.0.1.6
OUI version : 11.2.0.2.0
Log file location : /home/oracle/product/11.2/cfgtoollogs/opatch/opatch2012-06-20_16-53-36PM.log

The patch should be applied/rolled back in '-all_nodes' mode only.
Converting the RAC mode to '-all_nodes' mode.
Applying interim patch '10205230' to OH '/home/oracle/product/11.2'
Verifying environment and performing prerequisite checks...
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Backing up files...
Patching component oracle.rdbms, 11.2.0.2.0...
The patch will be installed on active database instances.
Installing and enabling the online patch 'bug10205230.pch', on database 'orcl01d1'.

Patching in all-node mode.
Updating nodes 'prod04' 
 Apply-related files are:
 FP = "/home/oracle/product/11.2/.patch_storage/10205230_Apr_2_2011_12_57_14/rac/copy_files.txt"
 DP = "/home/oracle/product/11.2/.patch_storage/10205230_Apr_2_2011_12_57_14/rac/copy_dirs.txt"
 MP = "/home/oracle/product/11.2/.patch_storage/10205230_Apr_2_2011_12_57_14/rac/make_cmds.txt"
 RC = "/home/oracle/product/11.2/.patch_storage/10205230_Apr_2_2011_12_57_14/rac/remote_cmds.txt"
Instantiating the file "/home/oracle/product/11.2/.patch_storage/10205230_Apr_2_2011_12_57_14/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/home/oracle/product/11.2/.patch_storage/10205230_Apr_2_2011_12_57_14/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file "/home/oracle/product/11.2/.patch_storage/10205230_Apr_2_2011_12_57_14/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/home/oracle/product/11.2/.patch_storage/10205230_Apr_2_2011_12_57_14/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Installing and enabling the online patch 'bug10205230.pch', on database 'orcl01d2' on node 'prod04'.
Patch 10205230 successfully applied
Log file location: /home/oracle/product/11.2/cfgtoollogs/opatch/opatch2012-06-20_16-53-36PM.log
OPatch succeeded.

You can verify that patch is applied and enabled

SQL> oradebug patch list
Patch File Name State
================ =========
bug10205230.pch ENABLED

If you have multiple ORACLE databases running out of same RDBMS Home, you can enable patch for other database using enableonlinepatch option

$ opatch util enableonlinepatch -connectString orcl02d1:sys:oracle123:prod03,orcl02d2:sys:oracle123:prod04 -id 10205230

In case of single instance home, below syntax is used

$ opatch util enableonlinepatch -connectString orcl:sys:oracle123 -id 10205230

Alert log is updated with following entries during patch application

Patch bug10205230.pch Installed - Update #1
Patch bug10205230.pch Enabled - Update #2
Thu Jun 21 02:13:17 2012
Online patch bug10205230.pch has been installed
Online patch bug10205230.pch has been enabled

Online patching is very important feature introduced in 11.2.0.2 as this allows you to apply patches without any downtime. Note that if you apply patch using normal opatch apply, it will not be enabled. You can confirm this by oradebug patch list command.

22 June, 2012 by 11g 1

Oracle Patch Set update (PSU 2) released for 10.2.0.4 and 11.1.0.7

Oracle has recently released Oracle Patch Set update (PSU 2) along with CPU October 2009. As informed earlier , PSU are released quarterly along with CPU patches.

If you are running 10.2.0.4 RAC database, you need to apply following patches

1) Apply Patch 8833280 (10.2.0.4.2 PSU 2) on Oracle Database 10.2.0.4
2) Apply Patch 8436582 (10.2.0.4 CRS Bundle Patch #4) on Oracle Clusterware 10.2.0.4
3) Apply Patch 8705958 (10.2.0.4 CRS PSU 2 ) on Oracle Clusterware 10.2.0.4

1)  Patch 8833280 (10.2.0.4.2 PSU 2) for Oracle Database 10.2.0.4

2) Patch 8436582 (10.2.0.4 CRS Bundle Patch #4) for Oracle Clusterware 10.2.0.4 CRS PSU2 patch contains fixes for bugs fixed in Bundle patch 4

3)  Patch 8705958 (10.2.0.4 CRS PSU 2 ) for Oracle Clusterware 10.2.0.4

Critical Patch Update (CPU) are released each quarter and contain security fixes identified by Oracle. Oracle has recently introduced, Patch Set Update (PSUs) which are proactive cumulative patches containing recommended bug fixes that are released on a regular and predictable schedule (it also contains Security fixes part of CPU). PSUs are on the same quarterly schedule as the Critical Patch Updates (CPU), specifically the Tuesday closest to the 15th of January, April, July, and October.

It is recommended to apply latest Database Patchet (aleady on 10.2.0.4) along with latest Patch Set Update (PSU2) to Oracle Database as this helps to fix the critical bugs and also serve as a new baseline version for reporting issues to Oracle.

In case of RAC, you also need to apply CRS Patch Bundle to fix the CRS issues identified by Oracle.

Please note that Patch Set Updates can be applied on the base release version or on any earlier Patch Set Update. For example, 10.2.0.4.2 can be applied on 10.2.0.4.1 and 10.2.0.4.0 . Once a Patch Set Update has been applied, the recommended way to get future security content is to apply subsequent Patch Set Updates. Reverting from an applied Patch Set Update back to the Critical Patch Update, while technically possible, requires significant time and effort, and is not advised.

To know further, please read following My Oracle Support articles

Note 405820.1 – 10.2.0.X CRS Bundle Patch Information

Note 854428.1 – Intro to Patch Set Updates (PSU)

Note 8833280.8 lists down the bugs fixed as part of PSU2.

29 October, 2009 by 10g 0

OUI-67124 – Copy failed from ‘location 1’ to ‘location 2’

Just a short note to discuss a problem faced by me while applying CPU Jan patch to clusterware on AIX 5L. I was getting following errors

UtilSession failed: ApplySession failed in system modification phase... 'ApplySession::apply failed: Copy failed from '/archive/oracle/soft/Patch/6980307/6756433/files/lib/libhasgen10.so' to '/oracle/crs_base/app/product/crs10gR2/lib/libhasgen10.so'...
Copy failed from '/archive/oracle/soft/Patch/6980307/6756433/files/lib/libocr10.so' to '/oracle/crs_base/app/product/crs10gR2/lib/libocr10.so'...
Copy failed from '/archive/oracle/soft/Patch/6980307/6756433/files/lib/libocrb10.so' to '/oracle/crs_base/app/product/crs10gR2/lib/libocrb10.so'...
Copy failed from '/archive/oracle/soft/Patch/6980307/6756433/files/lib/libocrutl10.so' to '/oracle/crs_base/app/product/crs10gR2/lib/libocrutl10.so'..

I had followed all the  pre-requsites for this patch installation i.e

1)Stopped the database instance and ASM instance on the node

2)Stopped the nodeapps services

3)Stopped the clusterware

4) Executed /usr/sbin/slibclean as root

I searched over metalink and found a note recommending renaming the files and  retrying the patching process. One more suggestion was to copy the files manually. I thought of debugging this issue (also wanted to have clean installation), so I checked for processes being run by ‘oracle’ user. I found that listener was running

oracle 1982506       1   0 00:30:13      -  0:00 /oracle/ora_base/app/product/db10gR2/bin/tnslsnr LISTENER_TAF_PRODDB1 -inherit

This was a listener which was created manually (not using netca) and not registered in the OCR. As a result, it did not stop when we stopped the nodeapps services. I then stopped the listener and executed /usr/sbin/slibclean (as root) and re-initiated the patching process. This time it went fine.

One more easier way would have been to use ‘fuser’ command to identify the pid’s for the processes accessing the file.

In the end I realized that before proceeding to apply patch ,it is better to check for if any Instance or listener or any other process (RMAN, sqlplus or sqlloader utilities too) is running from Oracle Home being patched even though you have followed all the steps mentioned in Patch readme.

Cheers

Amit

30 March, 2009 by 10g 0

CPU (Critical Patch Update) and N-apply

OVERVIEW

Oracle has been regularly releasing security fixes for all it’s products to take care of any security vulnerability. Since 2005 Oracle formulated a policy that it will release CPU patch ( bundle of security fixes) every quarter (January, April, July, and October).

CPU includes fixes for various Oracle products including Database. Initially Oracle decided to include security fixes, fixes that conflict with security fix and fixes for bugs that affect wide range of users.

With each CPU Oracle provides a brief overview of fixes included, but does not elaborate on the vulnerabilities as this information could be misused. CPU patches are applied using Opatch utility.

CONFLICTS – SUPER SET – SUBSET

Before moving any further let us understand what a patch conflict is. As we all know, Oracle is bunch of c-programs. Any patch makes one or more changes to the c program files. Two patches can make changes to the same or different sets of c-code. When a patch shares one or more files with an existing patch then it is known as CONFLICT.

If a old patch makes all the changes to the c-code as compared to the new patch plus a set of additional changes, then the old patch is the SUPERSET of new patch. On the contrary if an existing patch contains only some of the changes compared to the new patch being installed, then it is called a SUBSET. While apply any new patch, Opatch checks for these 3 conditions.

In case of a superset situation, Opatch does not install the new patch and in case of subset old patch is rolled back and new patch is installed. When opatch detects a conflict scenario, user can decide which patch to keep and which patch to rollback.

N-APPLY

While applying a cpu patch, users can face a patch conflict with an all ready applied patch. You can request Oracle for a merge patch. CPU patches are cumulative i.e. fixes included in Jan fix will be part of April fix.

To take care of all the conflicts reported by various users, Oracle releases CPU super merge patches mid-way between 2 CPU patch releases. CPU super merge patches are merge of a CPU patch and all the reported patches conflicting with a given CPU patch.

This entire process was time consuming. Conflicts also prevent application of CPU patches. To overcome this, Oracle has come up with a new concept of packaging cpu patches for 10.2.0.3 (not all platforms) and upwards since July 2007. With this new process Oracle has stopped including non-security fixes in CPUs. Oracle has come up with N-apply concept.

With the new N-apply concept, a cpu is divided into one or more molecules. Molecule is set of one or more bug fixes. This feature is specially use full when users encounter conflicts. Conflicts can be resolved at the molecular level. This provides us a technique to apply a part of cpu fixes while patch conflicts are resolved.

To make use of the n-apply feature ensure that you have latest version of opatch, run the command:

opatch napply -skip_subset -skip_duplicate

OPatch parameter -skip_subset, which is specified at CPU patch install, enables OPatch to determine that a merge patch that includes a molecule is already installed in the Oracle home, and to then skip application of the molecule patch.

OPatch parameter, -skip_duplicate, provides the additional benefit of detecting when a molecule patch has already been applied, as in the case of a previous CPU, and to skip application of it. This reduces the length of time required to do the n-apply CPU installation and minimizes the overall change to the Oracle home.

4 May, 2008 by 10g 1

Disclaimer!!

This blog reflect our own views and do not necessarily represent the views of our current or previous employers.

The contents of this blog are from our experience, you may use at your own risk, however you are strongly advised to cross reference with Product documentation and test before deploying to production environments.

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 188 other subscribers