Many times i came across a common problem in RAC databases where trying to add an asm disk is not possible due to errors like
ORA-15075 “disk(s) are not visible cluster-wide”
ORA-15020 “discovered duplicate ASM disk “DISK1″ and
ORA-15054 “disk “ORCL:DISK1″ does not exist in diskgroup “DG1″.
Rebalancing the diskgroup and trying to add the disk with “FORCE” option also does not help in this case.
I will be discussing how to come out of a situation like this i.e When you are trying to add an asm disk in cluster environment and it says that disk is already added and when trying to drop the same disk it says that disk is not present in the diskgroup.
Lets start from the very begining:
I have decided to add an asm disk in RAC environment to an already existing diskgroup DATA1.
Login to asm instance “/ as sysdba”
SQL > ALTER DISKGROUP DATA1 ADD DISK ‘/dev/rdsk/c1t2d3s4′;
But it failed with following error:
ALTER DISKGROUP DATA1 ADD DISK '/dev/rdsk/c1t2d3s4';* ERROR at line 1: <strong>ORA-15032: not all alterations performed ORA-15075: disk(s) are not visible cluster-wide</strong>
This is due to the fact that the physical disk partition is not visible from all RAC nodes. Then i contacted the sysadmins to make sure that the disk is visible from all RAC nodes and accessible by ORACLE. They have fixed the problem and now the disk /dev/rdsk/c1t2d3s4 can be seen from all RAC nodes. Then i tried to add the disk again using force option as:
SQL > ALTER DISKGROUP DATA1 ADD DISK ‘/dev/rdsk/c1t2d3s4′ force;
But it failed with following error:
ORA-15020: discovered duplicate ASM disk “/dev/rdsk/c1t2d3s4″
It shows that disk with same name is already present in the diskgroup.
As it shows that the disk is already present in the diskgroup, while trying to drop the disk i got following error:
SQL> alter diskgroup DATA1 drop disk '/dev/rdsk/c1t2d3s4'; alter diskgroup DATA1 drop disk '/dev/rdsk/c1t2d3s4' * ERROR at line 1: <strong>ORA-15032 : not all alterations performed ORA-15054 : disk "/dev/rdsk/c1t2d3s4" does not exist in diskgroup "DATA1"</strong>
Now I cannot move further as adding and dropping the disk is not possible here. Then I decided to check the status of the disk from v$asm_disk from all RAC nodes, to do this issue following query:
SQL > col name format a15 SQL > col path format a20 SQL > select GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,NAME,PATH from v$asm_disk;
We Obain following results from all the nodes :
G# D# HEADER_STATU MOUNT_S STATE NAME PATH
—- —- ———— ——- ——– ———— ————————-
0 0 MEMBER IGNORED NORMAL /dev/rdsk/c1t2d3s4
Header_status=MEMBER means that the disk is a valid asm disk on all RAC nodes.
Mount_status=IGNORED means that Disk is present in the system, but is ignored by ASM.
Group_number=0 This is the number used when a disk is not mounted by a diskgroup.
Now by checking the dd output of the disk as :
$dd if=/dev/rdsk/c1t2d3s4 of=/tmp/disk.out bs=4096 count=1096
$ vi /tmp/disk.out
I found that the diskgroup name and disk number allocated to this disk, which confirms that the disk is now a part of diskgroup DATA1.
But from the results of the header_status,mount_status and group_number it is clear that the disk is partially added to RAC asm instances. To correct this we will have to clear the disk header to add it again:
# dd if=/dev/null of=/dev/rdsk/c1t2d3s4 bs=4096 count=5000
This command cleared the disk header and after that disk was added successfully.
Note: - Please note that using dd will clear the ASM header and should be used only after confirming the disk. Using it on a wrong disk can cause Diskgroup to dismount and lead to Data Loss.
Hello,
Great this tip that you sent, you helped me so much.
but take care when to use dd command.
Regards,
I had a simillar ORA-15020: discovered duplicate ASM disk issue, our sysadmin determined that there was a BAD superblock on the disk, he newfs’ed them. Then I used the dd commands with if=/dev/zero, then I re-formatted the device (solaris 10 format command), then I was able to “ALTER DISKGROUP DATA ADD DISK ‘/dev/rdsk/emcpower49a’;”
I am facing similar issue in a Production System. To avoid using dd command, can we un-present and present the Disk using SAN admin module. Is it equivalant to using dd command?
Csk,
I am not sure about SAN admin module but unless you erase the disk header (using dd), you will not be able to add it back. You can work with oracle support and let them verify your issue.
Cheers
Amit
I am facing similar issue in Production System. But hesitating to use dd command. Is there any other safe way to do this?
Yeah. I have opened SR with Oracle. Once I get solution , I will provide it here.
Hi CSK,
In this case, the information is already written into disk header, and to reuse the disk the only option i s to erase the header contents and I think dd is a safe way only as we can verify before actually using dd.
I am not aware of any oracle command which will erase the disk header.
Please let us know what reply you get from Oracle Support on this!!
Regards,
Saurabh Sood
In my case the adding Disk to an existing Disk Group failed as the disk was not visible from other RAC node.Now the Disk is showing Header_Status as MEMBER and Mount_Status as CLOSED. dd command can be used to erase the Header Status of the Disk. But the Disk Group’s working Disk has details about the Disk that failed to get added. This is causing me worry…
I raised P1 SR and Oracle support checked the Disks belonging to the Diskgroup and asked me to unmount & mount the Diskgroups to make sure that the partially added disk has not done any serious damage. As I was able to unmount and mount the Diskgroups , they asked me to create a new Diskgroup with FORCE option and it worked. Problem got resolved. Didnt require to use dd command.
Thanks.
Just a note — I ran into this installing Grid on a standalone server, and it turns out there is a bug when using non-default discovery paths for ASM on standalone. Metalink note 1086140.1 goes into details and a workaround…
What was the version of Oracle 10gr2. As per Oracle support force option should work with 10.2.0.4. We have similar issue with one of our disk in 10.2.0.4 and opened a SR. I also gave reference of your link but they support person says that force option should work with 10.2.0.4.
Hi Rafiq,
I am not aware what was the version of my 10gr2 database, but the force usage behavior is not changed in these version. YOu can probably ask for any ORACLE BUG reference to this from Oracle Support.
As per ASM dcumentation the force should work in all 10gr2 version, but in my case it did not work at all.
let us know also if you get any reply from oracle support on any BUG related to this.
BTW did using “force” worked in your case?
Regards,
Saurabh Sood
Perfect, this article really helped me. I used dd if=/dev/zero. it worked.
Thanks a lot.
-Yogi