Before I start my post, I would like to wish all our readers a “Happy New Year”.
While reviewing a 10gR2 RAC configuration I faced following errors on invoking cluvfy utility
$ ./cluvfy stage -post crsinst -n prod01,prod02 -verbose Performing post-checks for cluster services setup Checking node reachability... Check: Node reachability from node "prod01-fe" Destination Node Reachable? ------------------------------------ ------------------------ prod01 yes prod02 yes Result: Node reachability check passed from node "prod01-fe". Checking user equivalence... Check: User equivalence for user "oracle" Node Name Comment ------------------------------------ ------------------------ prod02 passed prod01 passed Result: User equivalence check passed for user "oracle". ERROR: CRS is not installed on any of the nodes. Verification cannot proceed. Post-check for cluster services setup was unsuccessful on all the nodes.
Cluvfy has reported that Clusterware has not been installed on the server. But this was strange and unexpected as one RAC Database was already running on these nodes and crs_stat reported the status of all CRS resources. ocrcheck also did not report any problems with the OCR. To dig further I checked Cluvfy logs located under $ORA_CRS_HOME/cv/log. It had recorded following errors
[main] [15:52:41:800] [OUIData.readInventoryData:393] ==== CRS home added: Oracle home properties: Name : OraCRS Type : CRS-HOME Location : /app/oracle/product/10.2/crs Node list: [prod01-fe, prod02-fe] ; Thu Dec 24 15:52:41 GMT+08:00 2009 [main] [15:52:41:800] [OUIData.readInventoryData:401] ==== ORACLE home added: Oracle home properties: Name : OraHome Type : ORACLE-HOME Location : /app/oracle/product/10.2/db_1 Node list: [prod01, prod02] ; Thu Dec 24 15:52:41 GMT+08:00 2009 [main] [15:52:41:800] [VerificationUtil.isCRSInstalled:1262] CRS wasn't found installed on node: prod02; Thu Dec 24 15:52:41 GMT+08:00 2009 [main] [15:52:41:800] [VerificationUtil.isCRSInstalled:1262] CRS wasn't found installed on node: prod01; Thu Dec 24 15:52:41 GMT+08:00 2009
You can notice that CRS_HOME has recorded Node list as [prod01-fe, prod02-fe] and for DB_HOME it is [prod01, prod02]. Actual hostname for the nodes are prod01 and prod02. prod01-fe and prod02-fe were aliases for the two nodes.
Cluvfy uses Central Oracle Inventory to obtain ORACLE_HOME information. Checking the $ORACLE_BASE/oraInventory/ContentsXML/inventory.xml confirmed that the nodes corresponding to CRS_HOME were stored as prod01-fe and prod02-fe ( I have changed the Angled brackets to Round brackets as html considers them as tags. Don’t worry your inventory file is not corrupted 🙂 )
(INVENTORY)
(VERSION_INFO)
(SAVED_WITH)10.2.0.1.0(/SAVED_WITH)
(MINIMUM_VER)2.1.0.6.0(/MINIMUM_VER)
(/VERSION_INFO)
(HOME_LIST)
(HOME NAME="OraCRS" LOC="/app/oracle/product/10.2/crs" TYPE="O" IDX="1" CRS="true")
(NODE_LIST)
(NODE NAME="prod01-fe"/)
(NODE NAME="prod02-fe"/)
(/NODE_LIST)
(/HOME)
(HOME NAME="OraHome" LOC="/app/oracle/product/10.2/db_1" TYPE="O" IDX="2")
(NODE_LIST)
(NODE NAME="prod01"/)
(NODE NAME="prod02"/)
(/NODE_LIST)
(/HOME)
There are few reported issues on metalink but they relate to CRS=”true” not being present for CRS_HOME entry. There is one more reported issue listed here by Surachart which was caused by incorrect permissions on /etc/oraInst.loc file.
To correct this problem, we had to run runInstaller -updateNodelist command to update the correct nodes in file. Even though some metalink notes recommend to change this file manually, I would recommend using runInstaller command for updating inventory.
Before executing the command, I ran olsnodes command to confirm that OCR stored prod01,prod02 in its repository.
runInstaller -updateNodeList -silent "CLUSTER_NODES={prod01,prod02}" ORACLE_HOME="/app/oracle/product/10.2/crs" ORACLE_HOME_NAME="OraCRS" LOCAL_NODE="prod01" CRS=true
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /var/opt/oracle/oraInst.loc
The inventory is located at /app/oracle/oraInventory
'UpdateNodeList' was successful.
Now the cluvfy worked fine. I do not have any reasoning for why the incorrect nodes were recorded in Oracle Inventory, but above solution should take care of this.
Recent Comments