Today I faced a strange issue with CRS post host reboot. CRS was not coming up and we could see following message in $ORA_CRS_HOME/log/<hostname>/client/clsc*.log
cat clsc26.log Oracle Database 10g CRS Release 10.2.0.4.0 Production Copyright 1996, 2008 Oracle. All rights reserved. 2011-07-01 21:00:14.345: [ COMMCRS]clsc_connect: (0x6945e0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET)) 2011-07-01 21:00:14.345: [ COMMCRS]clsc_connect: (0x695020) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth))
It looked like like issue with socket files, so I removed /var/tmp/.oracle files (This is RHEL4 box). Tried starting crs with 'crsctl start crs' and still no socket files were written. /tmp/crsctl*log files were getting generated but they were empty. I spent close to 1 hour rebooting host and trying various stuff. Then I decided to run the daemons mentioned in /etc/inittab manually i.e
/etc/init.d/init.evmd run /etc/init.d/init.cssd fatal /etc/init.d/init.crsd run
When I ran init.evmd I got following errors
# /etc/init.d/init.evmd run Startup will be queued to init within 30 seconds. /home/oracle/.bash_profile: line 6: ulimit: open files: cannot modify limit: Operation not permitted *** glibc detected *** double free or corruption (fasttop): 0x0000000000688960 *** -bash: line 1: 17389 Aborted /apps/oracle/product/102crs/bin/crsctl check boot >/tmp/crsctl.17085
It pointed to issue with .bash_profile so I renamed it to .old and retried the operation. This time it succeeded and crs also came up fine.
There was entry for ulimit -n 2048 in .bash_profile which was causing it. I am not aware why ulimit is causing issue, will try to find it and post details