ORA-15063 – ASM Discovered Insufficient amount of Disks

Looking at the response generated by my earlier post on ASM, I thought of writing one more article.

Many of us (Working on ASM) would have encountered following errors during startup of ASM instance

ORA-15032: not all alterations performed
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA1"

Corresponding ASM alert log shows

ERROR: no PST quorum in group 1: required 2, found 1
NOTE: cache dismounting group 1/0x94F1292B (DATA1)
NOTE: dbwr not being msg'd to dismount
ERROR: diskgroup DATA1 was not mounted

Like any other error , first approach to solving any “ORA” error should be look at its description

<span style="color: #ff0000;">/home/oracle&gt;oerr ora 15063
15063, 00000, "ASM discovered an insufficient number of disks for diskgroup \"%s\""
// *Cause:  ASM was unable to find a sufficient number of disks belonging to the
//          diskgroup to continue the operation.
// *Action: Check that the disks in the diskgroup are present and functioning,
//          that the owner of the ORACLE binary has read/write permission to
//          the disks, and that the ASM_DISKSTRING initialization parameter
//          has been set correctly.  Verify that ASM discovers the appropriate
//          disks by querying V$ASM_DISK from the ASM instance.
</span>

Now if you see the explanation, it clearly mentions that there are some disks missing in diskgroup which is not allowing the diskgroup to be mounted. It also gives the Action plan to be taken

1)Check that the disks in the diskgroup are present and functioning

To check this you need to query V$ASM_DISK and see if all the disks are visible and also MOUNT_STATUS shows as Member

SQL> SELECT GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,STATE,NAME,PATH FROM V$ASM_DISK;

GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU STATE    NAME       PATH
------------ ----------- ------- ------------ -------- ---------- --------------------
1           0 CACHED  MEMBER       NORMAL   DATA_0000  /dev/sda7

In case the disks are not member, you will most likely see the GROUP_NUMBER corresponding to that disk as 0.

2)Check that Owner of the ORACLE binary has read/write permission to the disks, and that the ASM_DISKSTRING initialization parameter has been set correctly.

You need to check the permissions for the disk and ensure that Oracle is owner

<span style="color: #3333ff;">$ls -ltr /dev/sda7
brw-rw----  1 <strong>oracle</strong> oinstall 8, 7 May  4 18:43 /dev/sda7</span>

You can also use dd command to see if the disk is accessible by Oracle user.e.g

$ id
uid=100(oracle) gid=100(oinstall)
$ dd if=/dev/rdsk/1 of=/dev/null bs=1024 count=100
dd: /dev/rdsk/1: open: Invalid argument

It should have shown something like

100+0 in
100+0 out

In case you are using ASMLIB . then you can use following command to see if disks are visible

$/etc/init.d/oracleasm listdisks

If it does not display any disk, then check if Oracle is having correct permissions

ls -ltr /dev/oracleasm/disks/*

To Check ASM_DISKSTRING parameter you can check previous post on ASM Disk Discovery

3)Verify that ASM discovers the appropriate disks by querying V$ASM_DISK from the ASM instance.

This is again similar to point 2 as in case the permissions are not set or ASM_DISKSTRING is not correctly set, it will not recognize the disk in V$ASM_DISK.

In addition to this , we can also use Oracle utility called kfed to check the problematic disk and verify if disk is part of ASM diskgroup. Kfed is not part of standard oracle installation and has to be generated. Please find below steps for same

For 10.2 above

<span style="color: #3333ff;"><strong>$cd $ORACLE_HOME/rdbms/lib
$ make -f ins_rdbms.mk ikfed</strong> </span>

<span style="color: #3333ff;">
</span> <span style="color: #3333ff;">Linking KFED utility (kfed)
rm -f /u01/app/oracle/product/asm10.2/rdbms/lib/kfed
gcc -o /u01/app/oracle/product/asm10.2/rdbms/lib/kfed -L/u01/app/oracle/product/asm10.2/rdbms/lib/ -L/u01/app/oracle/product/asm10.2/lib/ -L/u01/app/oracle/product/asm10.2/lib/stubs/ -L/usr/lib -lirc /u01/app/oracle/product/asm10.2/lib/s0main.o /u01/app/oracle/product/asm10.2/rdbms/lib/sskfeded.o /u01/app/oracle/product/asm10.2/rdbms/lib/skfedpt.o /u01/app/oracle/product/asm10.2/rdbms/lib/defopt.o -ldbtools10 -lclntsh `cat /u01/app/oracle/product/asm10.2/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /u01/app/oracle/product/asm10.2/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lclient10 -lnnetd10 -lvsn10 -lcommon10 -lgeneric10 -lmm -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /u01/app/oracle/product/asm10.2/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /u01/app/oracle/product/asm10.2/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lclient10 -lnnetd10 -lvsn10 -lcommon10 -lgeneric10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lclient10 -lnnetd10 -lvsn10 -lcommon10 -lgeneric10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /u01/app/oracle/product/asm10.2/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/asm10.2/lib -lm `cat /u01/app/oracle/product/asm10.2/lib/sysliblist` -ldl -lm -L/u01/app/oracle/product/asm10.2/lib
mv -f /u01/app/oracle/product/asm10.2/bin/kfed /u01/app/oracle/product/asm10.2/bin/kfedO
mv: cannot stat `/u01/app/oracle/product/asm10.2/bin/kfed': No such file or directory
make: [ikfed] Error 1 (ignored)
mv /u01/app/oracle/product/asm10.2/rdbms/lib/kfed /u01/app/oracle/product/asm10.2/bin/kfed
chmod 751 /u01/app/oracle/product/asm10.2/bin/kfed</span>

Now to use kfed, we need to use synatx as kfed read devicename

<span style="color: #3333ff;">$ kfed read /dev/sda8
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj:              2147483648 ; 0x008: TYPE=0x8 NUMB=0x0
kfbh.check:                  2083552713 ; 0x00c: 0x7c307dc9
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:         ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]:            0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]:            0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
kfdhdb.compat:                168820736 ; 0x020: 0x0a100000
kfdhdb.dsknum:                        0 ; 0x024: 0x0000
<strong>kfdhdb.grptyp:                        1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER</strong>
</span> <strong><span style="color: #3333ff;">kfdhdb.dskname:               DATA_0000 ; 0x028: length=9
kfdhdb.grpname:                    DATA ; 0x048: length=4
kfdhdb.fgname:                DATA_0000 ; 0x068: length=9</span>
</strong>

I have made the important things as Bold (You can also say, things which I know. For rest need to check with Oracle Support 🙂 )
Lets verify the result with output from V$ASM_DISK and V$ASM_DISKGROUP

<span style="color: #3333ff;">select DG.GROUP_NUMBER "G.NO",DG.NAME,D.DISK_NUMBER,D.MOUNT_STATUS,
D.HEADER_STATUS,DG.TYPE,D.NAME,D.PATH FROM V$ASM_DISK D,V$ASM_DISKGROUP DG 
where DG.GROUP_NUMBER=D.GROUP_NUMBER;</span>

<span style="color: #3333ff;">
</span> <span style="color: #3333ff;"> G.NO NAME       DISK_NUMBER MOUNT_S HEADER_STATU TYPE   NAME       PATH
---------- ---------- ----------- ------- ------------ ------ ---------- ----------
1 DATA                 0 CACHED  MEMBER       EXTERN DATA_0000  /dev/sda8</span>

kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL ->This indicates Redundancy for Group.Check TYPE in query output.

kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER – > This indicates Disk Header status. Here it indicates it is member of Group.

kfdhdb.dskname: DATA_0000 ; 0x028: length=9 -> This indicates Disk Name
kfdhdb.grpname: DATA ; 0x048: length=4 ->This indicates the Group Name for the disk.

kfdhdb.fgname: DATA_0000 ; 0x068: length=9 ->This indicates the Failure Group Name.

Going back to error, we saw

ERROR: no PST quorum in group 1: required 2, found 1

Here PST means Partnership Status Table. PST contains list of disks(partners) which are required to mount the diskgroup. In case you are using External Redundancy and ASM is not able to find one disk , then the diskgroup will fail to mount.

You can read more about PST in book Oracle Automatic Storage Management: Under-the-Hood & Practical Deployment Guide (Oracle)
by Nitin Vengurlekar (Author), Murali Vallath (Author), Rich Long

11g Update on Kfed

You are not required to generate kfed on 11g installation. It is part of the default oracle installation.