cluvfy

Before I start my post, I would like to wish all our readers a “Happy New Year”.

While reviewing a 10gR2 RAC configuration I faced following errors on invoking cluvfy utility

$ ./cluvfy stage -post crsinst -n prod01,prod02 -verbose

Performing post-checks for cluster services setup 

Checking node reachability...

Check: Node reachability from node "prod01-fe"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  prod01                              yes
  prod02                              yes
Result: Node reachability check passed from node "prod01-fe".

Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Comment
  ------------------------------------  ------------------------
  prod02                              passed
  prod01                              passed
Result: User equivalence check passed for user "oracle".

ERROR:
CRS is not installed on any of the nodes.
Verification cannot proceed.

Post-check for cluster services setup was unsuccessful on all the nodes.

Cluvfy has reported that Clusterware has not been installed on the server. But this was strange and unexpected as one RAC Database was already running on these nodes and crs_stat reported the status of all CRS resources. ocrcheck also did not report any problems with the OCR. To dig further I checked Cluvfy logs located under $ORA_CRS_HOME/cv/log. It had recorded following errors

[main] [15:52:41:800] [OUIData.readInventoryData:393]  ==== CRS home added: Oracle home properties:
Name     : OraCRS
Type     : CRS-HOME
Location : /app/oracle/product/10.2/crs
Node list: [prod01-fe, prod02-fe]
; Thu Dec 24 15:52:41 GMT+08:00 2009
[main] [15:52:41:800] [OUIData.readInventoryData:401]  ==== ORACLE home added: Oracle home properties:
Name     : OraHome
Type     : ORACLE-HOME
Location : /app/oracle/product/10.2/db_1
Node list: [prod01, prod02]
; Thu Dec 24 15:52:41 GMT+08:00 2009
[main] [15:52:41:800] [VerificationUtil.isCRSInstalled:1262]  CRS wasn't found installed  on node: prod02; Thu Dec 24 15:52:41
 GMT+08:00 2009
[main] [15:52:41:800] [VerificationUtil.isCRSInstalled:1262]  CRS wasn't found installed  on node: prod01; Thu Dec 24 15:52:41
 GMT+08:00 2009

You can notice that CRS_HOME has recorded Node list as [prod01-fe, prod02-fe] and for DB_HOME it is [prod01, prod02]. Actual hostname for the nodes are prod01 and prod02. prod01-fe and prod02-fe were aliases for the two nodes.

Cluvfy uses Central Oracle Inventory to obtain ORACLE_HOME information. Checking the $ORACLE_BASE/oraInventory/ContentsXML/inventory.xml confirmed that the nodes corresponding to CRS_HOME were stored as prod01-fe and prod02-fe ( I have changed the Angled brackets to Round brackets as html considers them as tags. Don’t worry your inventory file is not corrupted 🙂 )

(INVENTORY)
(VERSION_INFO)
   (SAVED_WITH)10.2.0.1.0(/SAVED_WITH)
   (MINIMUM_VER)2.1.0.6.0(/MINIMUM_VER)
(/VERSION_INFO)
(HOME_LIST)
(HOME NAME="OraCRS" LOC="/app/oracle/product/10.2/crs" TYPE="O" IDX="1" CRS="true")
   (NODE_LIST)
      (NODE NAME="prod01-fe"/)
      (NODE NAME="prod02-fe"/)
   (/NODE_LIST)
(/HOME)
(HOME NAME="OraHome" LOC="/app/oracle/product/10.2/db_1" TYPE="O" IDX="2")
   (NODE_LIST)
      (NODE NAME="prod01"/)
      (NODE NAME="prod02"/)
   (/NODE_LIST)
(/HOME)

There are few reported issues on metalink but they relate to CRS=”true” not being present for CRS_HOME entry. There is one more reported issue listed here by Surachart which was caused by incorrect permissions on /etc/oraInst.loc file.

To correct this problem, we had to run runInstaller -updateNodelist command to update the correct nodes in file. Even though some metalink notes recommend to change this file manually,  I would recommend using runInstaller command for updating inventory.
Before executing the command, I ran olsnodes command to confirm that OCR stored prod01,prod02 in its repository.

runInstaller -updateNodeList -silent "CLUSTER_NODES={prod01,prod02}" ORACLE_HOME="/app/oracle/product/10.2/crs" ORACLE_HOME_NAME="OraCRS" LOCAL_NODE="prod01" CRS=true
Starting Oracle Universal Installer...

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /var/opt/oracle/oraInst.loc
The inventory is located at /app/oracle/oraInventory
'UpdateNodeList' was successful.

Now the cluvfy worked fine. I do not have any reasoning for why the incorrect nodes were recorded in Oracle Inventory, but above solution should take care of this.