11G: OCR Manual Backups

OCR is a file that manages cluster and Oracle Real Application Clusters (Oracle RAC) database configuration information. The OCR contains information about the cluster node list, instance-to-node mapping information, and information about Oracle Clusterware resource profiles for applications

This is a Very important component along with Voting Disk and Loss of it will require you to re-install the Clusterware Software.

Oracle 10g provides Automatic Backup of OCR along with manually export of OCR files.

Starting from Oracle 11g, we can also take manual backups of OCR. So in 11g , we can have following backups

1) Automatic backups—Oracle Clusterware automatically creates OCR backups every four hours. At any one time, Oracle always retains the last three backup copies of the OCR. The CRSD process that creates the backups also creates and retains an OCR backup for each full day and at the end of each week. You cannot customize the backup frequencies or the number of files that Oracle retains.

We can view the backups using ocrconfig -showbackup auto command.

[oracle@prod01]/home/oracle>ocrconfig -showbackup auto
prod01     2008/05/14 19:23:22     /u01/app/11.1.0/crs/cdata/prod_cluster/backup00.ocr
prod01     2008/05/14 15:23:21     /u01/app/11.1.0/crs/cdata/prod_cluster/backup01.ocr
prod01     2008/05/14 11:23:21     /u01/app/11.1.0/crs/cdata/prod_cluster/backup02.ocr
prod01     2008/05/13 15:23:20     /u01/app/11.1.0/crs/cdata/prod_cluster/day.ocr
prod01     2008/05/04 01:52:33     /u01/app/11.1.0/crs/cdata/prod_cluster/week.ocr

2)Manual backups—You can use the ocrconfig -manualbackup command to force Oracle Clusterware to perform a backup of the OCR at any time, rather than wait for the automatic backup that occurs at 4-hour intervals. The -manualbackup option is especially useful when you to need to obtain a binary backup on demand, such as before you make changes to the OCR.

You need to be logged in as root user to take manual backups. Use ocrconfig –manualbackup command. To view the backups use ocrconfig –showbackup manual command


[oracle@prod01]/home/oracle>ocrconfig -showbackup manual
prod01     2008/05/01 12:21:29     /u01/app/11.1.0/crs/cdata/prod_cluster/backup_20080501_122129.ocr
prod02     2008/04/11 11:32:50     /u01/app/11.1.0/crs/cdata/prod_cluster/backup_20080411_113250.ocr
prod02     2008/03/27 17:25:23     /u01/app/11.1.0/crs/cdata/prod_cluster/backup_20080327_172523.ocr

3)Manual Export—You can Use the ocrconfig -export file_name command to export the OCR content to a file format.

For more details refer to 11g Documentation

15 May, 2008 by 11g 0

ORA-07445 [opidsa()+480] – Metalink Alert for 10.2.0.3

Oracle has reported a new Alert for user’s who have upgraded their databases to  10.2.0.3 patchset on windows.

User processes dumps core dumps with following errors

ORA-07445: exception encountered: core dump [<strong>opidsa()</strong>+480]
[SIGSEGV] [Address not mapped to object] [0x000000000] [] []

or

ORA-07445: exception encountered: core dump [ACCESS_VIOLATION]
 [_opidsa+360] [PC:0x2080540] [ADDR:0x0] [UNABLE_TO_READ] []
 

Patch 5648872 can be applied to resolve the issue. For more information, refer to
Metalink Note:418531.1-ORA-7445[opidsa] after Applying the 10.2.0.3 Patch Set

This issue is fixed in 10.2.0.4 Patchset 

13 May, 2008 by 10g 0

Significance of LD_LIBRARY_PATH

Few days back, I wanted to remove a instance semaphore as they were not cleared automatically due to abnormal shutdown. As there were few other databases running on the server , so I thought of capturing the semaphore id to remove them manually using “ipcrm” .
For this purpose, we are required to use sysresv utility. Upon execution of the command, I received following errors

$/u01/app/oracle/product/10.2.0/bin>sysresv
sysresv: error while loading shared libraries: libclntsh.so.10.1: cannot open shared object file: No such file or directory

Now this was a new installation so I was bit scared if there was something wrong with the installation. Searching the web pointed me to articles which mentioned about using user which is not in correct group.
But in my case it was properly set and part of dba/oinstall group. Further research led me to Puschitz webpage (Very useful site if you are planning to install Database)which described that this problem is caused when environment variable LD_LIBRARY_PATH is not set.
So I set the value of LD_LIBRARY_PATH and then tried again

<strong>$</strong>export LD_LIBRARY_PATH=$ORACLE_HOME/lib
<strong>$</strong>sysresv
IPC Resources for ORACLE_SID "orcl10" :
Shared Memory:
ID KEY
131074 0x0291b1b8
Semaphores:
ID KEY
229377 0xc116e83c
262146 0xc116e83d
294915 0xc116e83e
Oracle Instance alive for sid "orcl10"

This worked !!!
My issue was solved but I must admit that I was not aware of what this variable did. I had seen this Environment variable set in my previous company, but somehow I didn’t give importance to it.
Searching on web, I came across Oracle Forum thread
which stated that
“The LD_LIBRARY_PATH environment variable contains a colon-separated list of directories in which the linker will search for library (.a) files. If you want the linker to search additional libraries, you can add their names to LD_LIBRARY_PATH.”

One more article gave insight into how application looks for library files.

http://docsun.cites.uiuc.edu/sun_docs/C/solaris_9/SUNWdev/LLM/p4.html
The runtime linker only looks in one default location for dependencies. This location is /usr/lib when processing 32-bit objects, and /usr/lib/64 when processing 64-bit objects. All other directories to be searched must be added to the runtime linker’s search path explicitly.
On searching metalink for further information on this parameter, I came across

Subject: Solaris: Setting LD_LIBRARY_PATH in 10gR1 and higher
Doc ID: Note:271451.1

According to this note, there are two more variables LD_LIBRARY_PATH_32 and LD_LIBRARY_PATH_64 which can be set to point to 32-Bit and 64-Bit libraries.
Further it mentioned that it is Not Recommended to set LD_LIBRARY_PATH unless application (Oracle or 3rd-party) requires it and recommended using LD_LIBRARY_PATH_32 and LD_LIBRARY_PATH_64.
This is done so that 32-Bit programs can use the correct libraries and similarly 64-bit programs can use 64-Bit library files.

So next time, you receive some library error, then do not panic 🙂 and see if parameter LD_LIBRARY_PATH or LD_LIBRARY_PATH_32 or LD_LIBRARY_PATH_64 is set.

Note:- In case you are using non-oracle user and see that you are getting same error after setting the variable,then check if the file ibclntsh.so.10.1exists and non-oracle user’s have read/execute permission over it. If no then relink Oracle using relink all command

12 May, 2008 by 10g 11g 0

ASM Disk Discovery

While creating ASM diskgroup or adding new disk to a existing diskgroup, Disk should be visible in V$ASM_DISK.

ASM discovers and examines the contents of all of the disks that are in the paths that you designated with values in the ASM_DISKSTRING initialization parameter.

As per Oracle Docs, Disk discovery also occurs when you:

– Run the ALTER DISKGROUP…ADD DISK and ALTER DISKGROUP…RESIZE DISK commands

– Query the V$ASM_DISKGROUP and V$ASM_DISK views

Note: – You should try to use V$ASM_DISK_STAT to get faster results as access to this view does not lead to Disk Discovery.

While creating ASM Diskgroup, Disk Discovery is the most common issue. This is mostly due to Disk permission issue or incorrect setting for ASM_DISKSTRING parameter.

So as to check this , you can use KFOD utility provided by Oracle.

This can be found in $ASM_HOME/bin directory. Help can be seen using

 oracle@asm]/home/oracle&gt; kfod help=y

<span style="font-size:small;"><span style="font-family:Times New Roman;">_asm_a/llow_only_raw_disks<span>              </span>KFOD allow only raw devices [_asm_allow_</span></span>

<span style="font-size:small;"><span style="font-family:Times New Roman;">only_raw_disks=TRUE/(FALSE)]</span></span>

<span style="font-size:small;"><span style="font-family:Times New Roman;">_asm_l/ibraries<span>         </span>ASM Libraries[_asm_libraries='lib1','lib2',...]</span></span>

<span style="font-size:small;"><span style="font-family:Times New Roman;">_asms/id<span>                </span>ASM Instance[_asmsid=sid]</span></span>

<span style="font-size:small;"><span style="font-family:Times New Roman;">a/sm_diskstring<span>         </span>ASM Diskstring [asm_diskstring='discoverystring', 'disco</span></span><span style="font-size:small;"><span style="font-family:Times New Roman;">verystring' ...]</span></span>

<span style="font-size:small;"><span style="font-family:Times New Roman;">d/isks<span>          </span>Disks to discover [disks=raw,asm,all]</span></span>

<span style="font-size:small;"><span style="font-family:Times New Roman;">g/roup<span>          </span>Group discover [group=controlfile]</span></span>

<span style="font-size:small;"><span style="font-family:Times New Roman;">n/ohdr<span>          </span>KFOD header suppression [nohdr=TRUE/(FALSE)]</span></span>

<span style="font-size:small;"><span style="font-family:Times New Roman;">o/p<span>             </span>KFOD options type [OP=DISKS/GROUPS/ALL]</span></span>

<span style="font-size:small;"><span style="font-family:Times New Roman;">p/file<span>          </span>ASM parameter file [pfile='parameterfile']</span></span>

<span style="font-size:small;"><span style="font-family:Times New Roman;">s/tatus<span>    </span><span>     </span>Include disk header status [status=TRUE/(FALSE)]</span></span>

<span style="font-size:small;"><span style="font-family:Times New Roman;">v/erbose<span>                </span>KFOD verbose errors [verbose=TRUE/(FALSE)]</span></span>

<span style="font-size:small;font-family:Times New Roman;">KFOD-01000: file not found</span>

 

To discover the disks , you need to use disks=all clause

 

[oracle@asm] /home/oracle&gt;kfod disks=all</span>
kfod disks=all
--------------------------------------------------------------------------------
 Disk          Size Path
==================================================================
   1:     345648 Mb /dev/sda7
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
==================================================================
     +ASM2 /u01/app/oracle/product/asm10.2
     +ASM1 /u01/app/oracle/product/asm10.2

As seen from help menu, we can use raw and asm as option. One more important parameter which can be used is asm_diskstring to make sure that problem is not with initialization parameter.

[oracle@asm]/home/oracle>kfod asm_diskstring='/raw/*' disks=all
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
====================================================================
     +ASM2 /u01/app/oracle/product/asm10.2
     +ASM1 /u01/app/oracle/product/asm10.2

By default asm_diskstring looks in all the directories.

9 May, 2008 by 10g 11g 2

CPU (Critical Patch Update) and N-apply

OVERVIEW

Oracle has been regularly releasing security fixes for all it’s products to take care of any security vulnerability. Since 2005 Oracle formulated a policy that it will release CPU patch ( bundle of security fixes) every quarter (January, April, July, and October).

CPU includes fixes for various Oracle products including Database. Initially Oracle decided to include security fixes, fixes that conflict with security fix and fixes for bugs that affect wide range of users.

With each CPU Oracle provides a brief overview of fixes included, but does not elaborate on the vulnerabilities as this information could be misused. CPU patches are applied using Opatch utility.

CONFLICTS – SUPER SET – SUBSET

Before moving any further let us understand what a patch conflict is. As we all know, Oracle is bunch of c-programs. Any patch makes one or more changes to the c program files. Two patches can make changes to the same or different sets of c-code. When a patch shares one or more files with an existing patch then it is known as CONFLICT.

If a old patch makes all the changes to the c-code as compared to the new patch plus a set of additional changes, then the old patch is the SUPERSET of new patch. On the contrary if an existing patch contains only some of the changes compared to the new patch being installed, then it is called a SUBSET. While apply any new patch, Opatch checks for these 3 conditions.

In case of a superset situation, Opatch does not install the new patch and in case of subset old patch is rolled back and new patch is installed. When opatch detects a conflict scenario, user can decide which patch to keep and which patch to rollback.

N-APPLY

While applying a cpu patch, users can face a patch conflict with an all ready applied patch. You can request Oracle for a merge patch. CPU patches are cumulative i.e. fixes included in Jan fix will be part of April fix.

To take care of all the conflicts reported by various users, Oracle releases CPU super merge patches mid-way between 2 CPU patch releases. CPU super merge patches are merge of a CPU patch and all the reported patches conflicting with a given CPU patch.

This entire process was time consuming. Conflicts also prevent application of CPU patches. To overcome this, Oracle has come up with a new concept of packaging cpu patches for 10.2.0.3 (not all platforms) and upwards since July 2007. With this new process Oracle has stopped including non-security fixes in CPUs. Oracle has come up with N-apply concept.

With the new N-apply concept, a cpu is divided into one or more molecules. Molecule is set of one or more bug fixes. This feature is specially use full when users encounter conflicts. Conflicts can be resolved at the molecular level. This provides us a technique to apply a part of cpu fixes while patch conflicts are resolved.

To make use of the n-apply feature ensure that you have latest version of opatch, run the command:

opatch napply -skip_subset -skip_duplicate

OPatch parameter -skip_subset, which is specified at CPU patch install, enables OPatch to determine that a merge patch that includes a molecule is already installed in the Oracle home, and to then skip application of the molecule patch.

OPatch parameter, -skip_duplicate, provides the additional benefit of detecting when a molecule patch has already been applied, as in the case of a previous CPU, and to skip application of it. This reduces the length of time required to do the n-apply CPU installation and minimizes the overall change to the Oracle home.

4 May, 2008 by 10g 1

Understanding of Oracle 11g Automatic Diagnostic Repository – Part 1

INTRODUCTION

In an endeavour to move towards self managed database and provide better support solutions, Oracle 11g introduces another new feature in 11g called the Automatic Diagnostic Repository or ADR. As the name implies ADR is a automatically managed file based repository of diagnostic information such as trace files, core dumps, alert log, etc… ADR is like a mini database managing traces of multiple instances and multiple prouducts (database, ASM, CRS, etc…) with metadata of these traces being maintained under a unified directory structure. ADR promises to use consistent diagnostic data structures across products. Set of tools bundled with ADR facilitate us in analyzing diagnostic data.

As an outcome of this feature:
– Critical errors captured in the diagnostic repository show up as alerts in EM. In cases where the issue is a known issue, Oracle recommends corrective actions to be implemented by DBA.
– It is intended to make it easy for DBAs to consolidate and transfer diagnostic data to Oracle support, thus facilitating faster resolution of the issue.

ARCHITECTURE

Starting with Oracle 11g R1, traditional *_DUMP_DEST parameters are ignored and the trace location is decided based on DIAGNOSTIC_DEST parameter. If this parameter is not set explicitly, it defaults to ORACLE_BASE environment variable if this is set, if this is not set, then it defaults to ORACLE_HOME/log.

DIAGNOSTIC_DEST is the ADR root directory also known as ADR base.

Each instance of each product stores diagnostic data in its own ADR home directory. For example, in a Real Application Clusters (RAC) environment with shared storage and ASM, each database instance and each ASM instance have a home directory within ADR.

Under ADR base are multiple ADR homes. ADR home is the root directory for all traces within an instance of a product. i.e. Each instance of a product has an ADR home. Directroy structure of ADR is pictorial represented as below:

Under the ADR base there will be a folder “diag”. Within this folder there will be seperate folders for each product like rdbsm, asm, crs, clients, lsnrctl, netcman, etc… Each of these folders will have seperate folders for every instance of the product.

For an instance of Oracle database, the structure is as depicted. All traces that used to be within USER_DUMP_DEST and BACKGROUND_DUMP_DEST are with trace folder. There is a seperate folder for alert log. A copy of the alert log in XML format is mainted in the alert folder.

Problem and Incident are two terms used frequently in ADR. All errors like the ora-600, ora-7445 or any other error like ora-470 that crashes database instance are termed as critical error. A critical error is termed as a Problem and is given a problem id by ADR based on it’s symptoms. Each occurrence of a critical error is termed as an Incident.

Incident directory contains multiple subdirectories, where each subdirectory is named for a particular incident, and where each contains dumps pertaining only to that incident.

Metadata directory contains ADR’s metadata.

HM directory contains health monitor reports which are part of ADR.

ADRCI – ADR command interpreter and EM support work bench are the two interfaces to ADR.

We will discuss these tools in our next post.

4 May, 2008 by 11g 0

Disclaimer!!

This blog reflect our own views and do not necessarily represent the views of our current or previous employers.

The contents of this blog are from our experience, you may use at your own risk, however you are strongly advised to cross reference with Product documentation and test before deploying to production environments.

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 188 other subscribers