[[email protected]]

Hi

I have a 2node RAC Setup 10gR2.

I have three ASM Volumes configured, out of devices sdb2, sdb3, sdb4

/etc/init.d/oracleasm createdisk VOL1 /dev/sdb2

/etc/init.d/oracleasm createdisk VOL2 /dev/sdb3

/etc/init.d/oracleasm createdisk VOL3 /dev/sdb4.

where VOL1 & VOL2 belong to the Disk Group ORCL_DATA1 and Vol3 to the group FLASH_RECOVERY_AREA.

I want to use the multipath disks namely /dev/multipath/sdb2, /dev/multipath/sdb3 and /dev/multipath/sdb4 for asm volumes.

I created the multipath disks on node1 and edited the file /etc/sysconfig/oracleasm and set ORACLEASM_SCANORDER=”multipath” and

ORACLEASM_SCANEXCLUDE=”sdb2 sdb3 sdb4″.

After a Reboot, the crs_stat shows

On Node1

---

Name Type Target State Host

---

ora….SM1.asm application ONLINE ONLINE linux1

ora….X1.lsnr application ONLINE ONLINE linux1

ora.linux1.gsd application ONLINE ONLINE linux1

ora.linux1.ons application ONLINE ONLINE linux1

ora.linux1.vip application ONLINE ONLINE linux1

ora….SM2.asm application ONLINE ONLINE linux2

ora….X2.lsnr application ONLINE ONLINE linux2

ora.linux2.gsd application ONLINE ONLINE linux2

ora.linux2.ons application ONLINE ONLINE linux2

ora.linux2.vip application ONLINE ONLINE linux2

ora.orcl.db application ONLINE ONLINE linux1

ora….l1.inst application ONLINE OFFLINE

ora….l2.inst application ONLINE ONLINE linux2

ora….test.cs application ONLINE ONLINE linux2

ora….cl1.srv application ONLINE OFFLINE

ora….cl2.srv application ONLINE ONLINE linux2

oracle@linux1 ~$ sqlplus “/ as sysdba”

SQL*Plus: Release 10.2.0.1.0 – Production on Tue Dec 8 12:04:53 2009

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to an idle instance.

SQL> select name, path from v$asm_disk;

select name, path from v$asm_disk

*

ERROR at line 1:

ORA-01034: ORACLE not available

I try to start the instance manually,

oracle@linux1 ~$ srvctl start instance -d orcl -i orcl1

PRKP-1001 : Error starting instance orcl1 on node linux1

CRS-0215: Could not start resource ‘ora.orcl.orcl1.inst’.

Also from sql

SQL> startup

ORA-01078: failure in processing system parameters

ORA-01565: error in identifying file ‘+ORCL_DATA1/orcl/spfileorcl.ora’

ORA-17503: ksfdopn:2 Failed to open file +ORCL_DATA1/orcl/spfileorcl.ora

ORA-15077: could not locate ASM instance serving a required diskgroup

On Node2

---

oracle@linux2 ~$ sqlplus “/ as sysdba”

SQL*Plus: Release 10.2.0.1.0 – Production on Tue Dec 8 11:56:13 2009

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to:

Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 – Production

With the Partitioning, Real Application Clusters, OLAP and Data Mining options

SQL> select name, path from v$asm_disk;

NAME

---

PATH

---

VOL1

ORCL:VOL1

VOL2

ORCL:VOL2

VOL3

ORCL:VOL3

Plz note, that I have the multipath devices only on one node currently.

I am new to the multipathing concept. Plz advise, on what (above) steps are wrong and what additional things are to be taken care of.

And the causes/solutions to the errors.

[Amit Bansal]

Hi Deepika,

I believe you are following below article available on Oracle website

http://www.oracle.com/technology/tech/linux/asmlib/multipath.html

First I believe that you should check ASM instance and see if you are able to see the disks on Node 1. From above output it looks that you connected to node 1 DB instance and not asm instance. This can also be seen from crs_stat output which shows that asm instance is up on node 1.

Secondly name multipath device as multipatha,multibathc instead of /multipath/sdb1 etc

[[email protected]]

Hi Amit,

Yes, I am following the above URL.

On Node1:

[root@linux1 ~]# /etc/init.d/oracleasm listdisks

VOL1

VOL2

VOL3

[root@linux1 ~]# /etc/init.d/oracleasm querydisk /dev/multipath/sdb2

Device “/dev/multipath/sdb2” is marked an ASM disk with the label “VOL1”

[root@linux1 ~]# /etc/init.d/oracleasm querydisk /dev/sdb2

Device “/dev/sdb2” is marked an ASM disk with the label “VOL1”

The O/P is same for sdb3 and sdb4 (both multipath & single path devices)

Is this what you mean when you say if disks are visible on node1?

[oracle@linux1 ~]$ srvctl status asm -n linux1

ASM instance +ASM1 is running on node linux1.

[oracle@linux1 ~]$ srvctl status instance -d orcl -i orcl1

Instance orcl1 is not running on node linux1

[oracle@linux1 ~]$ srvctl start instance -d orcl -i orcl1

PRKP-1001 : Error starting instance orcl1 on node linux1

CRS-0215: Could not start resource ‘ora.orcl.orcl1.inst’.

[oracle@linux1 ~]$

I am unable to start the instance orcl1 on node1.

Also what is the significance of naming multipath devices in a particular way?

[Amit Bansal]

Hi,

Try logging into ASM instance and query V$ASM_DISK i.e

export ORACLE_SID=+ASM1

sqlplus “/ as sysdba”

select name,path,mount_status,header_status from V$ASM_DISK;

with regards to naming convention, I think naming convention should follow /dev/[a-z]* naming convention instead of /dev/multipath/[a-z]*

It could be this or it could be that as the name of multipath device is also ending with sdb2,sdb3 and we are also excluding the disk ending with sdb2,sdb3. I am not too sure on this.

I think best way would be to actually first query V$ASM_DISK output.

[[email protected]]

Thanks Amit for the prompt reply.

I had ORACLE_SID SET to orcl1 and orcl2 on both the nodes respectively.

On node1, I did the above steps, and I luckily was able to connect to Oracle. It did not say ‘connected to idle instance’.

The O/P of the query on Node1 (ORACLE_SID=+ASM1) is

NAME PATH MOUNT_S HEADER_STATUS

ORCL:VOL1 CLOSED UNKNOWN

ORCL:VOL1 CLOSED UNKNOWN

ORCL:VOL1 CLOSED UNKNOWN

The O/P of the query on Node2 (ORACLE_SID=orcl2) is

NAME PATH MOUNT_S HEADER_STATUS

VOL1 ORCL:VOL1 OPENED UNKNOWN

VOL2 ORCL:VOL2 OPENED UNKNOWN

VOL3 ORCL:VOL3 OPENED UNKNOWN

Does that mean that the multipath disks have not been picked up by asm at all.

What about the permissions. I ran the foll commands (for sdb2, sdb3, sdb4) on Node1

# chown -R oracle:dba /dev/multipath/sdb2

# chmod -R 775 /dev/multipath/sdb2

# /etc/init.d/oracleasm scandisks

# /etc/init.d/oracleasm listdisks

VOL1

VOL2

VOL3

How do these permissions remain intact after a reboot to the node?

[Amit Bansal]

Deepika,

How have you configured multipathing? Are you using udev and device mapper?

Also let me know your linux release details.

-Amit

[[email protected]]

Hi Amit,

Actually I have an application that runs on every node on the cluster and creates the devices /dev/multipath/sd*. I am new to this entire setup, but I can tell you that it uses udev.

On Node 1

[oracle@linux1 ~]$ uname -a

Linux linux1 2.6.9-78.0.13.ELsmp #1 SMP Wed Jan 7 17:52:47 EST 2009 i686 i686 i386 GNU/Linux

Does this much information answer your questions?

[Amit Bansal]

Hi,

>>Actually I have an application that runs on every node on the cluster and creates the devices /dev/multipath/sd*

Can’t understand this statement.

Anyways if you use udev, you can set permissions in rules file

/etc/udev/rules.d/50-udev.rules

I think your sysadmin can help you with that.

-Amit

[[email protected]]

Amit,

I am finally able to achieve what I intended to do.

Many Thanks to you for your valuable suggestions.

[Author]

Posts