Tips

Cluster SSH tool Utility

Many times you come across scenario’s when you wish to open multiple ssh windows and execute same commands.e.g You wish to see alert log for multi-node RAC simultaneously or edit sysctl.conf files for multiple machines.

Cluster SSH utility helps solve this problem as it opens multiple SSH sessions and allows simultaneous control.

In case you are using MAC OS X, you can download from Google Code site . Utility is called csshX.

For Linux you can download from Sourceforge site and utility is called Cluster SSH.

I am using csshX to show demo. Suppose I have 3 hosts host1,host2,host3 I need to use following commands on terminal app

$csshX host1 host2 host3

or

$csshX host[1-3]

As you can see we can specify range using square brackets.  csshX will create an SSH session to each remote host in separate Terminal.app windows. A master window will also be created. All keyboard input in the master will be sent to all the slave windows. Below screenshot displays how windows will look like

I can enter commands in Master (Red Color) window and it will execute same commands in all 3 windows.If you want to execute commands in particular window then go directly  to that window.  In case we need to open 3 sessions for host1 then we execute

$csshx host1+3

Use awk/sed in vi

Thought of sharing some useful info which can help you to do your work faster in vi.You can use awk/sed scripts in vi using following command in vi

:%!scriptname

Here the scriptname file should have execute privilges for user. I used this to create a useful script which I was doing by typing multiple substitution command in vi.

e.g Your file contains list of table
$cat t.lst
BANK005
BJSTM
BJS_ORG
CHAINED_ROWS
CORR_BAM
CORR_CAM
CORR_EIT
CORR_GAC
CORR_GAM
CORR_ITC
CORR_LDT
CORR_LHT
Create script (quotes)  with following command and give execute permission to user.

sed -e “s/^/’/g” -e “s/$/’,/” $1|awk ‘{printf (“%s”,$0)}’|sed -e “s/^/(/g” -e “s/,$/)/g”

open t.lst in vi and type :%!quotes
('BANK005','BJSTM','BJS_ORG','CHAINED_ROWS','CORR_BAM','CORR_CAM','CORR_EIT','CORR_GAC','CORR_GAM','CORR_ITC','CORR_LDT','CORR_LHT')

Similarly if you wish to remove blank lines, have a file blank like

awk ‘!NF==0 {print $0}’ $1
Blank lines can also be directly removed from vi using :g/^$/d
Isn’t it cool.. 🙂

CRSCTL CheatSheet

You can find below various commands which can be used to administer Oracle Clusterware using crsctl. This is for purpose of easy reference.

Start Oracle Clusterware

#crsctl start crs

Stop Oracle Clusterware

#crsctl stop crs

Enable Oracle Clusterware

#crsctl enable crs

It enables automatic startup of Clusterware daemons

Disable Oracle Clusterware

#crsctl disable crs

It disables automatic startup of Clusterware daemons. This is useful when you are performing some
operations like OS patching and does not want clusterware to start the daemons automatically.

Checking Voting disk Location

$crsctl query css votedisk

0. 0 /dev/sda3
1. 0 /dev/sda5
2. 0 /dev/sda6
Located 3 voting disk(s).

Note: -Any command which just needs to query information can be run using oracle user. But anything which alters Oracle Clusterware requires root privileges.

Add Voting disk

#crsctl add css votedisk path

Remove Voting disk

#crsctl delete css votedisk path

Check CRS Status

$crsctl check crs

Cluster Synchronization Services appears healthy

Cluster Ready Services appears healthy

Event Manager appears healthy

You can also see particular daemon status

$crsctl check cssd

Cluster Synchronization Services appears healthy

$crsctl check crsd

Cluster Ready Services appears healthy

$crsctl check evmd

Event Manager appears healthy

You can also check Clusterware status on both the nodes using

$crsctl check cluster

prod01 ONLINE

prod02 ONLINE

Checking Oracle Clusterware Version

To determine software version (binary version of the software on a particular cluster node) use

$crsctl query crs softwareversion

Oracle Clusterware version on node [prod01] is [11.1.0.6.0]

For checking active version on cluster, use

$ crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [11.1.0.6.0]

As per documentation, multiple versions are used while upgrading.

There are other options for CRSCTL too which can be seen using

$crsctl

Or

$crsctl help

11.2 Reference

11.2 introduced few changes to crsctl usage. Most important is clusterized commands which allows you to perform remote operations. They are

  • crsctl check cluster
  • crsctl start cluster
  • crsctl stop cluster

All these commands allow following usage

Default Stop local server
-all Stop all servers
-n Stop named servers
server […] One or more blank-separated server names
-f Force option

Let’s see usage

% crsctl check cluster -all
**************************************************************
prod01:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
prod02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

crsctl pin css is used to associate node name with node number. i.e if olsnodes shows prod01 as 1, then it should persist. This is helpful if you intend to run pre 11.2 database

#crsctl pin css -n prod01
#crsctl pin css -n prod02

To check daemon status, following commands need to be used

Check crsd – crsctl check crs

Check cssd – crsctl check crs

check evmd – crsctl check evm

crs_unregister is replaced by crsctl delete resource <resource_name>

crs_stat has been deprecated (though still works) and you need to use

$crsctl stat res -t
e.g

--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       prod01
               ONLINE  ONLINE       prod02
ora.FLASH.dg
               ONLINE  ONLINE       prod01
               ONLINE  ONLINE       prod02
ora.LISTENER.lsnr
               ONLINE  ONLINE       prod01
               ONLINE  ONLINE       prod02
ora.asm
               ONLINE  ONLINE       prod01               Started
               ONLINE  ONLINE       prod02               Started
ora.gsd
               OFFLINE OFFLINE      prod01
               OFFLINE OFFLINE      prod02
ora.net1.network
               ONLINE  ONLINE       prod01
               ONLINE  ONLINE       prod02
ora.ons
               ONLINE  ONLINE       prod01
               ONLINE  ONLINE       prod02
ora.registry.acfs
               ONLINE  ONLINE       prod01
               ONLINE  ONLINE       prod02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       prod01
ora.cvu
      1        ONLINE  ONLINE       prod01
ora.oc4j
      1        ONLINE  ONLINE       prod01
ora.prod01.vip
      1        ONLINE  ONLINE       prod01
ora.prod02.vip
      1        ONLINE  ONLINE       prod02
ora.scan1.vip
      1        ONLINE  ONLINE       prod01
ora.tintin.db
      1        ONLINE  ONLINE       prod01               Open
      2        ONLINE  ONLINE       prod02               Open

Wrote following awk command to give output on one line

crsctl status res |grep -v "^$"|awk -F "=" 'BEGIN {print " "} {printf("%s",NR%4 ? $2"|" : $2"\n")}'|sed -e 's/  *, /,/g' -e 's/, /,/g'|\
awk -F "|" 'BEGIN { printf "%-40s%-35s%-20s%-50s\n","Resource Name","Resource Type","Target ","State" }{ split ($3,trg,",") split ($4,st,",")}{for (i in trg) {printf "%-40s%-35s%-20s%-50s\n",$1,$2,trg[i],st[i]}}'

output

Resource Name                           Resource Type                      Target              State                                             

ora.DATA.dg                             ora.diskgroup.type                 ONLINE              ONLINE on prod01                              
ora.DATA.dg                             ora.diskgroup.type                 ONLINE              ONLINE on prod02                              
ora.FLASH.dg                            ora.diskgroup.type                 ONLINE              ONLINE on prod01                              
ora.FLASH.dg                            ora.diskgroup.type                 ONLINE              ONLINE on prod02                              
ora.LISTENER.lsnr                       ora.listener.type                  ONLINE              ONLINE on prod01                              
ora.LISTENER.lsnr                       ora.listener.type                  ONLINE              ONLINE on prod02                              
ora.LISTENER_SCAN1.lsnr                 ora.scan_listener.type             ONLINE              ONLINE on prod01                              
ora.asm                                 ora.asm.type                       ONLINE              ONLINE on prod01                              
ora.asm                                 ora.asm.type                       ONLINE              ONLINE on prod02                              
ora.cvu                                 ora.cvu.type                       ONLINE              ONLINE on prod01                              
ora.gsd                                 ora.gsd.type                       OFFLINE             OFFLINE                                           
ora.gsd                                 ora.gsd.type                       OFFLINE             OFFLINE                                           
ora.net1.network                        ora.network.type                   ONLINE              ONLINE on prod01                              
ora.net1.network                        ora.network.type                   ONLINE              ONLINE on prod02                              
ora.oc4j                                ora.oc4j.type                      ONLINE              ONLINE on prod01                              
ora.ons                                 ora.ons.type                       ONLINE              ONLINE on prod01                              
ora.ons                                 ora.ons.type                       ONLINE              ONLINE on prod02                              
ora.prod01.vip                      ora.cluster_vip_net1.type          ONLINE              ONLINE on prod01                              
ora.prod02.vip                      ora.cluster_vip_net1.type          ONLINE              ONLINE on prod02                              
ora.registry.acfs                       ora.registry.acfs.type             ONLINE              ONLINE on prod01                              
ora.registry.acfs                       ora.registry.acfs.type             ONLINE              ONLINE on prod02                              
ora.scan1.vip                           ora.scan_vip.type                  ONLINE              ONLINE on prod01                              
ora.snowy.db                            ora.database.type                  OFFLINE             OFFLINE                                           
ora.snowy.db                            ora.database.type                  ONLINE              OFFLINE                                           
ora.tintin.db                           ora.database.type                  ONLINE              ONLINE on prod01                              
ora.tintin.db                           ora.database.type                  ONLINE              ONLINE on prod02                              
ora.tintin.tintin_db_svc.svc            ora.service.type                   ONLINE              ONLINE on prod02                              
ora.tintin.tintin_ggate_svc.svc         ora.service.type                   ONLINE              ONLINE on prod01