Comments on: How To Recover From Corrupted OCR Disk http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/ Writing About Our Experiences With Oracle Databases Thu, 09 Feb 2012 14:27:06 +0000 hourly 1 By: Saurabh Sood http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-7991 Saurabh Sood Fri, 22 Oct 2010 06:09:41 +0000 http://askdba.org/weblog/?p=253#comment-7991 Hi Kamal, I think the above steps should also work, but not sure about the correctness, have you ever tried these? YOu are correct about the listener configuration, listener is added through netca and is a part of nodeapps. This also clarifies Nitin's comment. Regards, Saurabh Sood Hi Kamal,

I think the above steps should also work, but not sure about the correctness, have you ever tried these?

YOu are correct about the listener configuration, listener is added through netca and is a part of nodeapps.

This also clarifies Nitin’s comment.

Regards,
Saurabh Sood

]]> By: Kamal http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-7909 Kamal Wed, 18 Aug 2010 12:36:29 +0000 http://askdba.org/weblog/?p=253#comment-7909 Hi Saurabh, Can we shorten all this exercise by following the steps. 1)Stop rdbms/asm/clusterware. 2)mv /etc/oracle to /etc/oracle.bak 3)dd out voting disk and ocr. 4)run root.sh on all nodes. 5)run vipca if required. Also i dont belive that listener will get added when you run vipca. Nodeapps will only register vip,gsd and ons. You have to manually add the listener as it has to run from RDBMS/ASM home and not from CRS home. Hi Saurabh,

Can we shorten all this exercise by following the steps.

1)Stop rdbms/asm/clusterware.
2)mv /etc/oracle to /etc/oracle.bak
3)dd out voting disk and ocr.
4)run root.sh on all nodes.
5)run vipca if required.

Also i dont belive that listener will get added when you run vipca. Nodeapps will only register vip,gsd and ons.
You have to manually add the listener as it has to run from RDBMS/ASM home and not from CRS home.

]]> By: bhallar http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-7790 bhallar Tue, 13 Apr 2010 15:13:05 +0000 http://askdba.org/weblog/?p=253#comment-7790 Jeffery Hunter published the same with some additional info. http://www.idevelopment.info/data/Oracle/DBA_tip/Oracle10gRAC/CLUSTER_70.shtml Jeffery Hunter published the same with some additional info.
http://www.idevelopment.info/data/Oracle/DBA_tip/Oracle10gRAC/CLUSTER_70.shtml

]]> By: Saurabh Sood http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-4728 Saurabh Sood Wed, 05 Aug 2009 09:28:52 +0000 http://askdba.org/weblog/?p=253#comment-4728 :) :)

]]> By: Bassem http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-4724 Bassem Wed, 05 Aug 2009 06:30:26 +0000 http://askdba.org/weblog/?p=253#comment-4724 great effor saurabh , thanks Bassem great effor saurabh , thanks
Bassem

]]> By: Saurabh Sood http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-4624 Saurabh Sood Thu, 30 Jul 2009 15:59:32 +0000 http://askdba.org/weblog/?p=253#comment-4624 Thanks Nitin for your comments. Amit: Thanks for your comments also. From 10g onwards listener is added as a nodeapps part. there is one command to configure listener with srvctl that can be seen in the link provided by Amit. There is no need to run netca again. With Regards, Saurabh Sood Thanks Nitin for your comments.

Amit: Thanks for your comments also.

From 10g onwards listener is added as a nodeapps part. there is one command to configure listener with srvctl that can be seen in the link provided by Amit.
There is no need to run netca again.

With Regards,
Saurabh Sood

]]> By: Amit http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-4622 Amit Thu, 30 Jul 2009 11:39:11 +0000 http://askdba.org/weblog/?p=253#comment-4622 Hi Nitin, This note is actually written by Saurabh and not by Emit :) Listener is registered with nodeapps so you need not register it again. Moreover there is no command like "srvctl add listener" . Refer http://download.oracle.com/docs/cd/B19306_01/rac.102/b14197/srvctladmin.htm#i1008403 If required you can change the default port for the listener using netca. Regards Amit Hi Nitin,

This note is actually written by Saurabh and not by Emit :)
Listener is registered with nodeapps so you need not register it again. Moreover there is no command like “srvctl add listener” . Refer http://download.oracle.com/docs/cd/B19306_01/rac.102/b14197/srvctladmin.htm#i1008403

If required you can change the default port for the listener using netca.

Regards
Amit

]]> By: Nitin http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-4604 Nitin Wed, 29 Jul 2009 07:27:37 +0000 http://askdba.org/weblog/?p=253#comment-4604 Hi Emit, Thanks for a well documented note. I think we are missing the netca step. netca must be used to register the listeners in the CRS. This has to be done after registering the database and instances using SRVCTL. Your thoughts on this please... Nitin R Hi Emit,

Thanks for a well documented note. I think we are missing the netca step. netca must be used to register the listeners in the CRS. This has to be done after registering the database and instances using SRVCTL. Your thoughts on this please…

Nitin R

]]> By: Amit http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-2249 Amit Sun, 29 Mar 2009 03:21:44 +0000 http://askdba.org/weblog/?p=253#comment-2249 Yes, you can also clean up rac installation and re-install the software. Then you can configure ASM instance and open the database. In fact I used the same approach for a Test cluster environment (in my case i messed up with crs bundle patch and installation got corrupted) Yes, you can also clean up rac installation and re-install the software. Then you can configure ASM instance and open the database. In fact I used the same approach for a Test cluster environment (in my case i messed up with crs bundle patch and installation got corrupted)

]]> By: Ming http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-2238 Ming Fri, 27 Mar 2009 14:45:51 +0000 http://askdba.org/weblog/?p=253#comment-2238 Thanks for note sharing. My question is whether those approaches are certified by Oracle? I am searching for a "back out" option before upgrading CRS from 10g to 11g and directed to your site. One step further to what you've pointed out, as a last resort, I guess we can completely re-install clusterware without losing the existing ASM and database. Would you agree? Ming Thanks for note sharing. My question is whether those approaches are certified by Oracle?
I am searching for a “back out” option before upgrading CRS from 10g to 11g and directed to your site. One step further to what you’ve pointed out, as a last resort, I guess we can completely re-install clusterware without losing the existing ASM and database. Would you agree?
Ming

]]> By: Saurabh Sood http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-1199 Saurabh Sood Wed, 31 Dec 2008 08:05:50 +0000 http://askdba.org/weblog/?p=253#comment-1199 Hi Suchi, I don't think there will be any data loss because we are shutting down all of the databases and the clusterware before performing these steps. Let me know if you have any queries. Cheers!!!! Saurabh Sood Hi Suchi,

I don’t think there will be any data loss because we are shutting down all of the databases and the clusterware before performing these steps.
Let me know if you have any queries.

Cheers!!!!
Saurabh Sood

]]> By: suchi http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-1198 suchi Wed, 31 Dec 2008 07:43:25 +0000 http://askdba.org/weblog/?p=253#comment-1198 Hi, good content... As I am new to RAC so don't have much idea but the way you descired here for OCR recover with this accpetance of loss of our data?or there is no loss...please suggest me. Thanks, Suchi Hi,

good content…
As I am new to RAC so don’t have much idea but the way you descired here for OCR recover with this accpetance of loss of our data?or there is no loss…please suggest me.

Thanks,
Suchi

]]> By: Saurabh Sood http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-658 Saurabh Sood Wed, 19 Nov 2008 05:53:46 +0000 http://askdba.org/weblog/?p=253#comment-658 Emit, The warning messages are ok, but could you please check whether the "crs" directory is present or not. $ ls -ltr /informat/u01/app/oracle/product/ You have to make sure that you already ran root.sh on the node from where rac installation was started and after that it is tried on other nodes. Regards, Saurabh Sood Emit,

The warning messages are ok, but could you please check whether the “crs” directory is present or not.
$ ls -ltr /informat/u01/app/oracle/product/

You have to make sure that you already ran root.sh on the node from where rac installation was started and after that it is tried on other nodes.

Regards,
Saurabh Sood

]]> By: Emit http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-648 Emit Tue, 18 Nov 2008 13:31:49 +0000 http://askdba.org/weblog/?p=253#comment-648 [root@sjum1blnx42 crs]# ./root.sh /bin/chmod: cannot access `/informat/u01/app/oracle/product/crs/evm/init': No such file or directory /bin/chmod: cannot access `/informat/u01/app/oracle/product/crs/css/init': No such file or directory /bin/chmod: cannot access `/informat/u01/app/oracle/product/crs/css/log': No such file or directory /bin/chmod: cannot access `/informat/u01/app/oracle/product/crs/css/auth': No such file or directory WARNING: directory '/informat/u01/app/oracle/product' is not owned by root WARNING: directory '/informat/u01/app/oracle' is not owned by root WARNING: directory '/informat/u01/app' is not owned by root WARNING: directory '/informat/u01' is not owned by root WARNING: directory '/informat' is not owned by root [root@sjum1blnx42 crs]# ./root.sh
/bin/chmod: cannot access `/informat/u01/app/oracle/product/crs/evm/init’: No such file or directory
/bin/chmod: cannot access `/informat/u01/app/oracle/product/crs/css/init’: No such file or directory
/bin/chmod: cannot access `/informat/u01/app/oracle/product/crs/css/log’: No such file or directory
/bin/chmod: cannot access `/informat/u01/app/oracle/product/crs/css/auth’: No such file or directory
WARNING: directory ‘/informat/u01/app/oracle/product’ is not owned by root
WARNING: directory ‘/informat/u01/app/oracle’ is not owned by root
WARNING: directory ‘/informat/u01/app’ is not owned by root
WARNING: directory ‘/informat/u01′ is not owned by root
WARNING: directory ‘/informat’ is not owned by root

]]> By: Saurabh Sood http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-644 Saurabh Sood Tue, 18 Nov 2008 04:15:23 +0000 http://askdba.org/weblog/?p=253#comment-644 Hi Emit, Please let us know what is the error that you got while running root.sh on other node. Regards Saurabh Hi Emit,

Please let us know what is the error that you got while running root.sh on other node.

Regards
Saurabh

]]> By: Emit http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-573 Emit Fri, 14 Nov 2008 18:45:05 +0000 http://askdba.org/weblog/?p=253#comment-573 Friend. I am facing a corrup OCR, i follow the recomedations below but i am stucked on the next step ==> Run root.sh for the other nodes. Looks like the second node do not have the image mount point of the node 1. Did you have any document to see more in deep. ? Regards Friend.

I am facing a corrup OCR, i follow the recomedations below but i am stucked on the next step ==> Run root.sh for the other nodes.

Looks like the second node do not have the image mount point of the node 1.

Did you have any document to see more in deep. ?

Regards

]]> By: Saurabh Sood http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-213 Saurabh Sood Fri, 12 Sep 2008 15:45:13 +0000 http://askdba.org/weblog/?p=253#comment-213 Dan, I have seen many situations where DBAs faced this issue. You are lucky that your installation have never been through corruption. I did not do anything special to introduce OCR corruption in my system, but still found the cluster in this state. I completely agree with you that we have automatic backups and mirroring of OCR disk but under certain conditions like : We have recently added node and before next backup (Before 4 hrs) we found OCR corruption, which leaves us with no recent backup or a scenario where some shell scripts wrongly deleted backups. I have listed down steps to recover when we do not have any backup available and we need to rebuild from scratch, which "as mentioned" requires downtime. In case we have backups available then definitely “ocrconfig -restore” is the way to go!! Cheers!!! Saurabh Sood Dan,

I have seen many situations where DBAs faced this issue. You are lucky that your installation have never been through corruption. I did not do anything special to introduce OCR corruption in my system, but still found the cluster in this state.

I completely agree with you that we have automatic backups and mirroring of OCR disk but under certain conditions like : We have recently added node and before next backup (Before 4 hrs) we found OCR corruption, which leaves us with no recent backup or a scenario where some shell scripts wrongly deleted backups.

I have listed down steps to recover when we do not have any backup available and we need to rebuild from scratch, which “as mentioned” requires downtime. In case we have backups available then definitely “ocrconfig -restore” is the way to go!!

Cheers!!!
Saurabh Sood

]]> By: Dan Norris http://askdba.org/weblog/2008/09/how-to-recover-from-corrupted-ocr-disk/#comment-211 Dan Norris Fri, 12 Sep 2008 01:56:42 +0000 http://askdba.org/weblog/?p=253#comment-211 "It is very common where a DBA is left with corrupted OCR disk without having any good backup." I think that is a false statement. First, it isn't common that OCR becomes corrupted. I've installed lots of clusters (even a significant number on virtual machines) and I've never had an OCR become corrupted unless I was intentionally trying to corrupt it. Secondly, backups are almost always available because Clusterware makes them automatically in $ORA_CRS_HOME/cdata/ and keeps several recent backups while rotating off the oldest ones. This process is described in the 10g R2 Clusterware and RAC deployment and admin guide, chapter 3. Your solution is much more drastic than required. The "ocrconfig -restore" command restores the OCR from a backup. I'd think that should be attempted first before blowing away the entire OCR contents and having to rebuild everything from scratch--a much more risky operation than simply restoring it. “It is very common where a DBA is left with corrupted OCR disk without having any good backup.”

I think that is a false statement. First, it isn’t common that OCR becomes corrupted. I’ve installed lots of clusters (even a significant number on virtual machines) and I’ve never had an OCR become corrupted unless I was intentionally trying to corrupt it. Secondly, backups are almost always available because Clusterware makes them automatically in $ORA_CRS_HOME/cdata/ and keeps several recent backups while rotating off the oldest ones. This process is described in the 10g R2 Clusterware and RAC deployment and admin guide, chapter 3.

Your solution is much more drastic than required. The “ocrconfig -restore” command restores the OCR from a backup. I’d think that should be attempted first before blowing away the entire OCR contents and having to rebuild everything from scratch–a much more risky operation than simply restoring it.

]]>