ASM and Multipathing

Home Forums Automatic Storage Management ASM and Multipathing

Tagged: ,

This topic contains 8 replies, has 0 voices, and was last updated by  deepika.dhamija@gmail.com 7 years, 3 months ago.

  • Author
    Posts
  • #1907

    Hi

    I have a 2node RAC Setup 10gR2.

    I have three ASM Volumes configured, out of devices sdb2, sdb3, sdb4

    /etc/init.d/oracleasm createdisk VOL1 /dev/sdb2

    /etc/init.d/oracleasm createdisk VOL2 /dev/sdb3

    /etc/init.d/oracleasm createdisk VOL3 /dev/sdb4.

    where VOL1 & VOL2 belong to the Disk Group ORCL_DATA1 and Vol3 to the group FLASH_RECOVERY_AREA.

    I want to use the multipath disks namely /dev/multipath/sdb2, /dev/multipath/sdb3 and /dev/multipath/sdb4 for asm volumes.

    I created the multipath disks on node1 and edited the file /etc/sysconfig/oracleasm and set ORACLEASM_SCANORDER=”multipath” and

    ORACLEASM_SCANEXCLUDE=”sdb2 sdb3 sdb4″.

    After a Reboot, the crs_stat shows

    On Node1


    Name Type Target State Host


    ora….SM1.asm application ONLINE ONLINE linux1

    ora….X1.lsnr application ONLINE ONLINE linux1

    ora.linux1.gsd application ONLINE ONLINE linux1

    ora.linux1.ons application ONLINE ONLINE linux1

    ora.linux1.vip application ONLINE ONLINE linux1

    ora….SM2.asm application ONLINE ONLINE linux2

    ora….X2.lsnr application ONLINE ONLINE linux2

    ora.linux2.gsd application ONLINE ONLINE linux2

    ora.linux2.ons application ONLINE ONLINE linux2

    ora.linux2.vip application ONLINE ONLINE linux2

    ora.orcl.db application ONLINE ONLINE linux1

    ora….l1.inst application ONLINE OFFLINE

    ora….l2.inst application ONLINE ONLINE linux2

    ora….test.cs application ONLINE ONLINE linux2

    ora….cl1.srv application ONLINE OFFLINE

    ora….cl2.srv application ONLINE ONLINE linux2

    oracle@linux1 ~$ sqlplus “/ as sysdba”

    SQL*Plus: Release 10.2.0.1.0 – Production on Tue Dec 8 12:04:53 2009

    Copyright (c) 1982, 2005, Oracle. All rights reserved.

    Connected to an idle instance.

    SQL> select name, path from v$asm_disk;

    select name, path from v$asm_disk

    *

    ERROR at line 1:

    ORA-01034: ORACLE not available

    I try to start the instance manually,

    oracle@linux1 ~$ srvctl start instance -d orcl -i orcl1

    PRKP-1001 : Error starting instance orcl1 on node linux1

    CRS-0215: Could not start resource ‘ora.orcl.orcl1.inst’.

    Also from sql

    SQL> startup

    ORA-01078: failure in processing system parameters

    ORA-01565: error in identifying file ‘+ORCL_DATA1/orcl/spfileorcl.ora’

    ORA-17503: ksfdopn:2 Failed to open file +ORCL_DATA1/orcl/spfileorcl.ora

    ORA-15077: could not locate ASM instance serving a required diskgroup

    On Node2


    oracle@linux2 ~$ sqlplus “/ as sysdba”

    SQL*Plus: Release 10.2.0.1.0 – Production on Tue Dec 8 11:56:13 2009

    Copyright (c) 1982, 2005, Oracle. All rights reserved.

    Connected to:

    Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 – Production

    With the Partitioning, Real Application Clusters, OLAP and Data Mining options

    SQL> select name, path from v$asm_disk;

    NAME


    PATH


    VOL1

    ORCL:VOL1

    VOL2

    ORCL:VOL2

    VOL3

    ORCL:VOL3

    Plz note, that I have the multipath devices only on one node currently.

    I am new to the multipathing concept. Plz advise, on what (above) steps are wrong and what additional things are to be taken care of.

    And the causes/solutions to the errors.

  • #2075

    Amit Bansal
    Keymaster

    Hi Deepika,

    I believe you are following below article available on Oracle website

    http://www.oracle.com/technology/tech/linux/asmlib/multipath.html

    First I believe that you should check ASM instance and see if you are able to see the disks on Node 1. From above output it looks that you connected to node 1 DB instance and not asm instance. This can also be seen from crs_stat output which shows that asm instance is up on node 1.

    Secondly name multipath device as multipatha,multibathc instead of /multipath/sdb1 etc

  • #2076

    Hi Amit,

    Yes, I am following the above URL.

    On Node1:

    [root@linux1 ~]# /etc/init.d/oracleasm listdisks

    VOL1

    VOL2

    VOL3

    [root@linux1 ~]# /etc/init.d/oracleasm querydisk /dev/multipath/sdb2

    Device “/dev/multipath/sdb2” is marked an ASM disk with the label “VOL1”

    [root@linux1 ~]# /etc/init.d/oracleasm querydisk /dev/sdb2

    Device “/dev/sdb2” is marked an ASM disk with the label “VOL1”

    The O/P is same for sdb3 and sdb4 (both multipath & single path devices)

    Is this what you mean when you say if disks are visible on node1?

    [oracle@linux1 ~]$ srvctl status asm -n linux1

    ASM instance +ASM1 is running on node linux1.

    [oracle@linux1 ~]$ srvctl status instance -d orcl -i orcl1

    Instance orcl1 is not running on node linux1

    [oracle@linux1 ~]$ srvctl start instance -d orcl -i orcl1

    PRKP-1001 : Error starting instance orcl1 on node linux1

    CRS-0215: Could not start resource ‘ora.orcl.orcl1.inst’.

    [oracle@linux1 ~]$

    I am unable to start the instance orcl1 on node1.

    Also what is the significance of naming multipath devices in a particular way?

  • #2077

    Amit Bansal
    Keymaster

    Hi,

    Try logging into ASM instance and query V$ASM_DISK i.e

    export ORACLE_SID=+ASM1

    sqlplus “/ as sysdba”

    select name,path,mount_status,header_status from V$ASM_DISK;

    with regards to naming convention, I think naming convention should follow /dev/[a-z]* naming convention instead of /dev/multipath/[a-z]*

    It could be this or it could be that as the name of multipath device is also ending with sdb2,sdb3 and we are also excluding the disk ending with sdb2,sdb3. I am not too sure on this.

    I think best way would be to actually first query V$ASM_DISK output.

  • #2078

    Thanks Amit for the prompt reply.

    I had ORACLE_SID SET to orcl1 and orcl2 on both the nodes respectively.

    On node1, I did the above steps, and I luckily was able to connect to Oracle. It did not say ‘connected to idle instance’.

    The O/P of the query on Node1 (ORACLE_SID=+ASM1) is

    NAME PATH MOUNT_S HEADER_STATUS

    ORCL:VOL1 CLOSED UNKNOWN

    ORCL:VOL1 CLOSED UNKNOWN

    ORCL:VOL1 CLOSED UNKNOWN

    The O/P of the query on Node2 (ORACLE_SID=orcl2) is

    NAME PATH MOUNT_S HEADER_STATUS

    VOL1 ORCL:VOL1 OPENED UNKNOWN

    VOL2 ORCL:VOL2 OPENED UNKNOWN

    VOL3 ORCL:VOL3 OPENED UNKNOWN

    Does that mean that the multipath disks have not been picked up by asm at all.

    What about the permissions. I ran the foll commands (for sdb2, sdb3, sdb4) on Node1

    # chown -R oracle:dba /dev/multipath/sdb2

    # chmod -R 775 /dev/multipath/sdb2

    # /etc/init.d/oracleasm scandisks

    # /etc/init.d/oracleasm listdisks

    VOL1

    VOL2

    VOL3

    How do these permissions remain intact after a reboot to the node?

  • #2079

    Amit Bansal
    Keymaster

    Deepika,

    How have you configured multipathing? Are you using udev and device mapper?

    Also let me know your linux release details.

    -Amit

  • #2080

    Hi Amit,

    Actually I have an application that runs on every node on the cluster and creates the devices /dev/multipath/sd*. I am new to this entire setup, but I can tell you that it uses udev.

    On Node 1

    [oracle@linux1 ~]$ uname -a

    Linux linux1 2.6.9-78.0.13.ELsmp #1 SMP Wed Jan 7 17:52:47 EST 2009 i686 i686 i386 GNU/Linux

    Does this much information answer your questions?

  • #2081

    Amit Bansal
    Keymaster

    Hi,

    >>Actually I have an application that runs on every node on the cluster and creates the devices /dev/multipath/sd*

    Can’t understand this statement.

    Anyways if you use udev, you can set permissions in rules file

    /etc/udev/rules.d/50-udev.rules

    I think your sysadmin can help you with that.

    -Amit

  • #2082

    Amit,

    I am finally able to achieve what I intended to do.

    Many Thanks to you for your valuable suggestions.

You must be logged in to reply to this topic.