Renaming Diskgroup containing Voting Disk and OCR

Recently encountered a issue where we had to rename a diskgroup for 11.2 RAC containing OCR and Voting Disk.At this moment we had not created database .

I had done similar activity on 11.2 Single instance having database files (see this post) using renamedg. Therefore we tried using same approach to do this activity.

Pre-requisite for renamedg is that diskgroup should be dismounted. As this Diskgroup was storing OCR/Voting disk, it was not possible to dismount the diskgroup. So we stopped the cluster on both the nodes and tried running renamdg command

Old Diskgroup name – DATA
New Diskgroup name – DG_DATA01

[oracle@prod-001]~% renamedg phase=both dgname=DATA newdgname=DG_DATA01 config=/oracle/renamedg.out asm_diskstring='ORCL:DISK*' verbose=true keep_voting_files=true

Parsing parameters..

Parameters in effect:

 	 Old DG name       : DATA
	 New DG name          : DG_DATA01
	 Phases               :
	 	 Phase 1
	 	 Phase 2
	 Discovery str        : ORCL:DISK*
	 Keep_voting_files  : TRUE
	 Clean              : TRUE
	 Raw only           : TRUE
renamedg operation: phase=both dgname=DATA newdgname=DG_DATA01 config=/oracle/renamedg.out asm_diskstring=ORCL:DISK* verbose=true keep_voting_files=true
Executing phase 1
Discovering the group
Performing discovery with string:ORCL:DISK*
Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK1 with disk number:0 and timestamp (32941163 273325056)
Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK2 with disk number:1 and timestamp (32941163 273325056)
Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK3 with disk number:2 and timestamp (32941163 273325056)
Checking for hearbeat...
Re-discovering the group
Performing discovery with string:ORCL:DISK*
Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK1 with disk number:0 and timestamp (32941163 273325056)
Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK2 with disk number:1 and timestamp (32941163 273325056)
Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK3 with disk number:2 and timestamp (32941163 273325056)
Checking if the diskgroup is mounted
Checking disk number:0
Checking disk number:1
Checking disk number:2
KFNDG-00600: file not found; arguments: [600] [Failed to initialize CSS context
] []
LEM-00031: Error encountered in lempgmh after calling lmserr.
KFNDG-00600: file not found; arguments: [600] [Failed to initialize CSS context
] []

Terminating kgfd context 0x2b592002f0a0

As we can see it failed informing that CSS is not up. Then we tried renamedg command by starting cluster (ASM Diskgroup was mounted) but it too failed with error

Checking if the diskgroup is mounted
Checking disk number:0
KFNDG-00405: file not found; arguments: [DATA]

Description of KFNDG clearly informs that diskgroup needs to be dismounted

KFNDG-00405: specified disk group string appears to be mounted
Cause: Disk group was mounted.
Action: Unmount the disk group and retry renamed

At this moment we thought that we cannot proceed and we would be required to re-install clusterware. Thinking over problem, I thought of moving the ocr and voting disk to different diskgroup and then renaming the old diskgroup.

Checked documentation and found out that there was command for replacing voting disk. In 11gR2 you can run this command while your cluster is up and running. So we created a new diskgroup DG_DATA01 and used following command

[oracle@prod-001]/dev/oracleasm/disks% crsctl replace votedisk +DG_DATA01
Failed to create voting files on disk group DG_DATA01.
Change to configuration failed, but was successfully rolled back.
CRS-4000: Command Replace failed, or completed with errors.

Oops. This failed. Checking ASM alert log gave the exact cause of this error. As you can see , it is asking for compatible.asm to be set to 11.2

NOTE: Creating voting files in diskgroup DG_DATA01
NOTE: Voting File refresh pending for group 2/0x88b6d50 (DG_DATA01)
NOTE: Attempting voting file creation in diskgroup DG_DATA01
ERROR: Voting file allocation failed for group DG_DATA01
Errors in file /oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_4272.trc:
ORA-15221: ASM operation requires compatible.asm of 11.2.0.0.0 or higher

Checking the compatability confirmed that it is set to 10.1

SQL> select name,COMPATIBILITY,DATABASE_COMPATIBILITY from V$ASM_DISKGROUP where name='DG_DATA01';

NAME       COMPATIBILITY      DATABASE_COMPATIBILITY
--------  ----------------   --------------------------
DG_DATA01  10.1.0.0.0        10.1.0.0.0

You can use set attribute to change the compatability. e.g

SQL> alter diskgroup DG_DATA01 SET ATTRIBUTE 'compatible.asm'='11.2';

Diskgroup altered.

Retrying the command succeeded

[oracle@prod-001]/dev/oracleasm/disks% crsctl replace votedisk +DG_DATA01
Successful addition of voting disk 241b1e0a36344f7bbfaca4a576d514e9                                                                                                                                                                                                                                .
Successful deletion of voting disk 72e47e5e3afb4fe9bfb502b1b4340503.
Successfully replaced voting disk group with +DG_DATA01.
CRS-4266: Voting file(s) successfully replaced

We verified same using below command

[root@prod-001 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   241b1e0a36344f7bbfaca4a576d514e9 (ORCL:DISK4) [DG_DATA01]
Located 1 voting disk(s).

This is external redundancy diskgroup and oracle allows only one voting disk in external redundancy diskgroup.

Next was to change ocr location. This can be done using ocrconfig which needs to be run as root user. We tried using replace command but it didn’t work as it was the only OCR configured on cluster.

[root@prod-001 ~]# ocrconfig -replace +DATA -replacement +DG_DATA01
PROT-28: Cannot delete or replace the only configured Oracle Cluster Registry location

So we added additional OCR disk to new diskgroup and dropped old one

[root@prod-001 ~]# ocrconfig -add +DG_DATA01
[root@prod-001 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
	 Version                  :          3
	 Total space (kbytes)     :     262120
	 Used space (kbytes)      :       2424
	 Available space (kbytes) :     259696
	 ID                       :  579998313
	 Device/File Name         :      +DATA
                                    Device/File integrity check succeeded
	 Device/File Name         : +DG_DATA01
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

	 Cluster registry integrity check succeeded

	 Logical corruption check succeeded

[root@prod-001 ~]# ocrconfig -delete +DATA
[root@prod-001 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
	 Version                  :          3
	 Total space (kbytes)     :     262120
	 Used space (kbytes)      :       2424
	 Available space (kbytes) :     259696
	 ID                       :  579998313
	 Device/File Name         : +DG_DATA01
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

	 Cluster registry integrity check succeeded

	 Logical corruption check succeeded

We then tried dropping diskgroup DATA

SQL> drop diskgroup data including contents;
drop diskgroup data including contents
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15027: active use of diskgroup "DATA" precludes its dismount

It failed as ASM instance was using the parameter file which was kept on ASM diskgroup. So created a new pfile and then re-created spfile on new Diskgroup

SQL> create pfile from spfile;
File created.

SQL> create spfile='+DG_DATA01' from pfile;

File created.

To drop the diskgroup , you will need to restart the cluster on both the nodes as ASM instance is still using the file. Run following commands on both the nodes

crsctl stop cluster
crsctl start cluster

Verify that OCR location is pointing to the correct diksgroup on all nodes before starting cluster. We faced a issue in one more environment, where for some reason OCR location was not properly updated on second node. In that case we modified it manually (after confirming OCR location using ocrcheck)

[root@prod-002 ~]# cat /etc/oracle/ocr.loc
#Device/file +DATA getting replaced by device +DG_DATA01
ocrconfig_loc=+DG_DATA01
local_only=false

You can verify the spfile and drop the old diskgroup now

SQL> sho parameter spfile

NAME				     TYPE			       VALUE
------------------------------------ --------------------------------- ------------------------------
spfile				     string			       +DG_DATA01/prod-cluster/asmp
								       arameterfile/registry.253.7290
								       69769
SQL> drop diskgroup data including contents;

Diskgroup dropped.

In case you don’t wish to drop this diskgroup as it contains datafiles and would like to rename the diskgroup, you can refer to my earlier post 11gR2: Steps to Rename ASM Diskgroup with DB files

Tags: 11gR2ASMoracle