In this blog post I will be discussing listener startup issues faced in 11gR2 RAC. I will be constantly updating this post based on my experiences or any comments on this blog post.
Let’s get started.You will experience following errors while starting listener using srvctl
[oracle@prod01]~% srvctl start listener -n prod01 PRCR-1013 : Failed to start resource ora.LISTENER.lsnr PRCR-1064 : Failed to start resource ora.LISTENER.lsnr on node prod01 CRS-5016: Process "/oragrid/product/11.2.0.2/bin/lsnrctl" spawned by agent "/oragrid/product/11.2.0.2/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/home/oragrid/product/11.2.0.2/log/prod01/agent/crsd/oraagent_oracle/oraagent_oracle.log" CRS-5016: Process "/oragrid/product/11.2.0.2/bin/lsnrctl" spawned by agent "/oragrid/product/11.2.0.2/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/home/oragrid/product/11.2.0.2/log/prod01/agent/crsd/oraagent_oracle/oraagent_oracle.log" CRS-2674: Start of 'ora.LISTENER.lsnr' on 'prod01' failed
Issue 1: – Incorrect ORACLE_HOME entry in listener.ora
This issue can be verified by attempting to start listener by lsnrctl utility. Please note that you need to use $GRID_HOME/bin/lsnrctl utility to manage listener in 11gR2 RAC
[oracle@prod01]~% lsnrctl start LSNRCTL for Linux: Version 11.2.0.2.0 - Production on 31-JAN-2012 08:09:52 Copyright (c) 1991, 2010, Oracle. All rights reserved. Starting /oragrid/product/11.2.0.2/bin/tnslsnr: please wait... TNSLSNR for Linux: Version 11.2.0.2.0 - Production System parameter file is /oragrid/product/11.2.0.2/network/admin/listener.ora Log messages written to /oracle/diag/tnslsnr/prod01/listener/alert/log.xml Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.160)(PORT=1521))) TNS-01201: Listener cannot find executable /oracle/product/11.2/bin/oracle for SID orcl01 Listener failed to start. See the error message(s) above..
Above error indicates that it is not able to find ‘oracle’ executable in specified path. But this is not our correct ORACLE_HOME. Checking listener.ora we found that this home is coming from ORACLE_HOME variable
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(SID_NAME =orcl01 )
(ORACLE_HOME = /oracle/product/11.2)
)
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = /oracle/product/11.2)
(PROGRAM = extproc)
)
)
Correct ORACLE_HOME is oracle/product/11.2.0.2 which needs to be updated correctly in listener.ora. After adding we were able to start the listener.
Issue 2: CRS resource ora.[node_name].ons is down
While debugging listener startup issue, you found that resource ora.[node_name].ons is not starting. Listener (including SCAN_LISTENER) is dependent on ora.ons resource
Checking $GRID_HOME/opmn/logs/ons.log[node_name], we see following messages
12/01/12 05:32:48 [ons-listener] Could not get address information for localhost 6100. 12/01/12 05:32:49 [internal] getaddrinfo(localhost, 6100, 1) failed (Name or service not known): 12/01/12 05:33:05 [internal] getaddrinfo(localhost, 6100, 1) failed (Name or service not known): 12/01/12 05:33:15 [internal] getaddrinfo(localhost, 6100, 1) failed (Name or service not known): 12/01/12 05:33:15 [internal] getaddrinfo(localhost, 6100, 1) failed (Name or service not known): 12/01/12 05:33:15 [internal] getaddrinfo(localhost, 6100, 1) failed (Name or service not known):
Issue is that your host is not able to resolve localhost setting.You can verify this by issuing ping localhost command.This issue can be resolved by adding following entry in /etc/hosts or to DNS
127.0.0.1 localhost
Issue 3: VIP and ora.[node_name].ons are not starting on one node
In this issue check that Bcast and Mask settings for Public interface are same on all nodes.
e.g ifconfig eth0 inet addr:192.168.1.4 Bcast:192.168.0.255 Mask:255.255.255.0
In our case we found that second node had Mask settings of 255.255.254.0. Correcting it and restarting interface resolved the issue.
