Lost all voting disk on ASM in oracle 11gr2

Home Forums Real Application Cluster Lost all voting disk on ASM in oracle 11gr2

This topic contains 3 replies, has 0 voices, and was last updated by  Pankaj Sharma 5 years, 4 months ago.

  • Author
    Posts
  • #2011

    Pankaj Sharma
    Participant

    Hi,

    This is regarding Oracle 11gR2 RAC. I have a 2 node RAC on RHEL5. I have my voting file on ASM diskgroup named “+DATA1”, which is created on device “/dev/sdc1”. Below is the details:

    [[email protected] ~]# oracleasm querydisk -p ASMDISK2

    Disk “ASMDISK2” is a valid ASM disk

    /dev/sdc1: LABEL=”ASMDISK2″ TYPE=”oracleasm”

    I was just want to test the scenario when I lost my all voting disk, then what happens actually. So I make the zero space the partition using below command:

    [[email protected] ~]# dd if=/dev/zero of=/dev/sdc1

    dd: writing to `/dev/sdc1′: No space left on device

    10474318+0 records in

    10474317+0 records out

    5362850304 bytes (5.4 GB) copied, 161.053 seconds, 33.3 MB/s

    My disktimeout was 200 seconds. But I waited for approx 8-10 minutes. Then I query the votedisk, but still it is showing ONLINE on DATA1 diskgroup.

    [[email protected] ~]# crsctl query css votedisk

    ## STATE File Universal Id File Name Disk group





    1. ONLINE 06d8300bb6af4fd3bfa2e1e5cda9c440 (ORCL:ASMDISK2) [DATA1]

    Located 1 voting disk(s).

    Also below are some findings in this situation:

    1) ASMCA is not showing the diskgroup DATA1, it is deleted.

    2) Cluster alert log is not displaying any message regarding the lost of voting disk.

    3) cssd.log also not displaying any message regarding the lost of voting disk.

    4) Cluster is still running and up. Also its working properly.

    As per documentation, oracle 11g r2 provide “rebootless node fencing” in case of all voting disk failure. But atleast it must display any message in log files or how the “crsctl query css votedisk” is still giving that voting disk is ONLINE ?

    I am not able to understand this situation. Please explain me the reason ?

    Thanks in advanced.

  • #2167

    Saurabh Sood
    Member

    Hi Pankaj,

    Can you provide the output of following command to me:

    $ kfed read /dev/sdc1

    Thanks,

    Saurabh Sood

  • #2168

    Amit Bansal
    Keymaster

    Pankaj,

    Is this a normal redundancy diskgroup?

    Regards

    Amit

  • #2169

    Pankaj Sharma
    Participant

    Hi,

    Sorry for the late reply.

    Amit:- This is external redundancy group.

    Saurabh:- That time I recovered the vote disk by restoring it. But just to get answer of your question, I have simulate the failure again, so that we can find out the reason for this unexpected behaviour of clusterware.

    This time my votedisk was on ASMDISK3 and corresponding os deveice is “/dev/sdd1”

    [BEFORE FAILURE]

    Below is the output from command “$ kfed read /dev/sdd1”

    [[email protected] ~]# kfed read /dev/sdd1

    kfbh.endian: 1 ; 0x000: 0x01

    kfbh.hard: 130 ; 0x001: 0x82

    kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD

    kfbh.datfmt: 1 ; 0x003: 0x01

    kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0

    kfbh.block.obj: 2147483648 ; 0x008: TYPE=0x8 NUMB=0x0

    kfbh.check: 711410310 ; 0x00c: 0x2a674286

    kfbh.fcn.base: 0 ; 0x010: 0x00000000

    kfbh.fcn.wrap: 0 ; 0x014: 0x00000000

    kfbh.spare1: 0 ; 0x018: 0x00000000

    kfbh.spare2: 0 ; 0x01c: 0x00000000

    kfdhdb.driver.provstr: ORCLDISKASMDISK3 ; 0x000: length=16

    kfdhdb.driver.reserved[0]: 1145918273 ; 0x008: 0x444d5341

    kfdhdb.driver.reserved[1]: 860574537 ; 0x00c: 0x334b5349

    kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000

    kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000

    kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000

    kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000

    kfdhdb.compat: 186646528 ; 0x020: 0x0b200000

    kfdhdb.dsknum: 0 ; 0x024: 0x0000

    kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL

    kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER

    kfdhdb.dskname: ASMDISK3 ; 0x028: length=8

    kfdhdb.grpname: DATA2 ; 0x048: length=5

    kfdhdb.fgname: ASMDISK3 ; 0x068: length=8

    kfdhdb.capname: ; 0x088: length=0

    kfdhdb.crestmp.hi: 32961169 ; 0x0a8: HOUR=0x11 DAYS=0x14 MNTH=0xc YEAR=0x7db

    kfdhdb.crestmp.lo: 252092416 ; 0x0ac: USEC=0x0 MSEC=0x1a8 SECS=0x30 MINS=0x3

    kfdhdb.mntstmp.hi: 32961387 ; 0x0b0: HOUR=0xb DAYS=0x1b MNTH=0xc YEAR=0x7db

    kfdhdb.mntstmp.lo: 2314231808 ; 0x0b4: USEC=0x0 MSEC=0x18 SECS=0x1f MINS=0x22

    kfdhdb.secsize: 512 ; 0x0b8: 0x0200

    kfdhdb.blksize: 4096 ; 0x0ba: 0x1000

    kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000

    kfdhdb.mfact: 113792 ; 0x0c0: 0x0001bc80

    kfdhdb.dsksize: 5114 ; 0x0c4: 0x000013fa

    kfdhdb.pmcnt: 2 ; 0x0c8: 0x00000002

    kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001

    kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002

    kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002

    kfdhdb.redomirrors[0]: 0 ; 0x0d8: 0x0000

    kfdhdb.redomirrors[1]: 0 ; 0x0da: 0x0000

    kfdhdb.redomirrors[2]: 0 ; 0x0dc: 0x0000

    kfdhdb.redomirrors[3]: 0 ; 0x0de: 0x0000

    kfdhdb.dbcompat: 168820736 ; 0x0e0: 0x0a100000

    kfdhdb.grpstmp.hi: 32961169 ; 0x0e4: HOUR=0x11 DAYS=0x14 MNTH=0xc YEAR=0x7db

    kfdhdb.grpstmp.lo: 251921408 ; 0x0e8: USEC=0x0 MSEC=0x101 SECS=0x30 MINS=0x3

    kfdhdb.vfstart: 96 ; 0x0ec: 0x00000060

    kfdhdb.vfend: 128 ; 0x0f0: 0x00000080

    kfdhdb.spfile: 0 ; 0x0f4: 0x00000000

    kfdhdb.spfflg: 0 ; 0x0f8: 0x00000000

    kfdhdb.ub4spare[0]: 0 ; 0x0fc: 0x00000000

    kfdhdb.ub4spare[1]: 0 ; 0x100: 0x00000000

    kfdhdb.ub4spare[2]: 0 ; 0x104: 0x00000000

    kfdhdb.ub4spare[3]: 0 ; 0x108: 0x00000000

    kfdhdb.ub4spare[4]: 0 ; 0x10c: 0x00000000

    kfdhdb.ub4spare[5]: 0 ; 0x110: 0x00000000

    kfdhdb.ub4spare[6]: 0 ; 0x114: 0x00000000

    kfdhdb.ub4spare[7]: 0 ; 0x118: 0x00000000

    kfdhdb.ub4spare[8]: 0 ; 0x11c: 0x00000000

    kfdhdb.ub4spare[9]: 0 ; 0x120: 0x00000000

    kfdhdb.ub4spare[10]: 0 ; 0x124: 0x00000000

    kfdhdb.ub4spare[11]: 0 ; 0x128: 0x00000000

    kfdhdb.ub4spare[12]: 0 ; 0x12c: 0x00000000

    kfdhdb.ub4spare[13]: 0 ; 0x130: 0x00000000

    kfdhdb.ub4spare[14]: 0 ; 0x134: 0x00000000

    kfdhdb.ub4spare[15]: 0 ; 0x138: 0x00000000

    kfdhdb.ub4spare[16]: 0 ; 0x13c: 0x00000000

    kfdhdb.ub4spare[17]: 0 ; 0x140: 0x00000000

    kfdhdb.ub4spare[18]: 0 ; 0x144: 0x00000000

    kfdhdb.ub4spare[19]: 0 ; 0x148: 0x00000000

    kfdhdb.ub4spare[20]: 0 ; 0x14c: 0x00000000

    kfdhdb.ub4spare[21]: 0 ; 0x150: 0x00000000

    kfdhdb.ub4spare[22]: 0 ; 0x154: 0x00000000

    kfdhdb.ub4spare[23]: 0 ; 0x158: 0x00000000

    kfdhdb.ub4spare[24]: 0 ; 0x15c: 0x00000000

    kfdhdb.ub4spare[25]: 0 ; 0x160: 0x00000000

    kfdhdb.ub4spare[26]: 0 ; 0x164: 0x00000000

    kfdhdb.ub4spare[27]: 0 ; 0x168: 0x00000000

    kfdhdb.ub4spare[28]: 0 ; 0x16c: 0x00000000

    kfdhdb.ub4spare[29]: 0 ; 0x170: 0x00000000

    kfdhdb.ub4spare[30]: 0 ; 0x174: 0x00000000

    kfdhdb.ub4spare[31]: 0 ; 0x178: 0x00000000

    kfdhdb.ub4spare[32]: 0 ; 0x17c: 0x00000000

    kfdhdb.ub4spare[33]: 0 ; 0x180: 0x00000000

    kfdhdb.ub4spare[34]: 0 ; 0x184: 0x00000000

    kfdhdb.ub4spare[35]: 0 ; 0x188: 0x00000000

    kfdhdb.ub4spare[36]: 0 ; 0x18c: 0x00000000

    kfdhdb.ub4spare[37]: 0 ; 0x190: 0x00000000

    kfdhdb.ub4spare[38]: 0 ; 0x194: 0x00000000

    kfdhdb.ub4spare[39]: 0 ; 0x198: 0x00000000

    kfdhdb.ub4spare[40]: 0 ; 0x19c: 0x00000000

    kfdhdb.ub4spare[41]: 0 ; 0x1a0: 0x00000000

    kfdhdb.ub4spare[42]: 0 ; 0x1a4: 0x00000000

    kfdhdb.ub4spare[43]: 0 ; 0x1a8: 0x00000000

    kfdhdb.ub4spare[44]: 0 ; 0x1ac: 0x00000000

    kfdhdb.ub4spare[45]: 0 ; 0x1b0: 0x00000000

    kfdhdb.ub4spare[46]: 0 ; 0x1b4: 0x00000000

    kfdhdb.ub4spare[47]: 0 ; 0x1b8: 0x00000000

    kfdhdb.ub4spare[48]: 0 ; 0x1bc: 0x00000000

    kfdhdb.ub4spare[49]: 0 ; 0x1c0: 0x00000000

    kfdhdb.ub4spare[50]: 0 ; 0x1c4: 0x00000000

    kfdhdb.ub4spare[51]: 0 ; 0x1c8: 0x00000000

    kfdhdb.ub4spare[52]: 0 ; 0x1cc: 0x00000000

    kfdhdb.ub4spare[53]: 0 ; 0x1d0: 0x00000000

    kfdhdb.acdb.aba.seq: 0 ; 0x1d4: 0x00000000

    kfdhdb.acdb.aba.blk: 0 ; 0x1d8: 0x00000000

    kfdhdb.acdb.ents: 0 ; 0x1dc: 0x0000

    kfdhdb.acdb.ub2spare: 0 ; 0x1de: 0x0000

    [[email protected] ~]#

    [AFTER FAILURE]

    [[email protected] ~]# kfed read /dev/sdd1

    kfbh.endian: 0 ; 0x000: 0x00

    kfbh.hard: 0 ; 0x001: 0x00

    kfbh.type: 0 ; 0x002: KFBTYP_INVALID

    kfbh.datfmt: 0 ; 0x003: 0x00

    kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0

    kfbh.block.obj: 0 ; 0x008: TYPE=0x0 NUMB=0x0

    kfbh.check: 0 ; 0x00c: 0x00000000

    kfbh.fcn.base: 0 ; 0x010: 0x00000000

    kfbh.fcn.wrap: 0 ; 0x014: 0x00000000

    kfbh.spare1: 0 ; 0x018: 0x00000000

    kfbh.spare2: 0 ; 0x01c: 0x00000000

    ERROR!!!, failed to get the oracore error message

    [[email protected] ~]# oracleasm querydisk -p ASMDISK3

    Disk “ASMDISK3” defines an unmarked device

    Still the same findings as before like no error message, vote disk are still ONLINE.

    Let me know if you want any other information.

    Thanks

    Pankaj

You must be logged in to reply to this topic.