Recently encountered a issue where we had to rename a diskgroup for 11.2 RAC containing OCR and Voting Disk.At this moment we had not created database .
I had done similar activity on 11.2 Single instance having database files (see this post) using renamedg. Therefore we tried using same approach to do this activity.
Pre-requisite for renamedg is that diskgroup should be dismounted. As this Diskgroup was storing OCR/Voting disk, it was not possible to dismount the diskgroup. So we stopped the cluster on both the nodes and tried running renamdg command
Old Diskgroup name – DATA
New Diskgroup name – DG_DATA01
[oracle@prod-001]~% renamedg phase=both dgname=DATA newdgname=DG_DATA01 config=/oracle/renamedg.out asm_diskstring='ORCL:DISK*' verbose=true keep_voting_files=true Parsing parameters.. Parameters in effect: Old DG name : DATA New DG name : DG_DATA01 Phases : Phase 1 Phase 2 Discovery str : ORCL:DISK* Keep_voting_files : TRUE Clean : TRUE Raw only : TRUE renamedg operation: phase=both dgname=DATA newdgname=DG_DATA01 config=/oracle/renamedg.out asm_diskstring=ORCL:DISK* verbose=true keep_voting_files=true Executing phase 1 Discovering the group Performing discovery with string:ORCL:DISK* Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK1 with disk number:0 and timestamp (32941163 273325056) Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK2 with disk number:1 and timestamp (32941163 273325056) Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK3 with disk number:2 and timestamp (32941163 273325056) Checking for hearbeat... Re-discovering the group Performing discovery with string:ORCL:DISK* Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK1 with disk number:0 and timestamp (32941163 273325056) Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK2 with disk number:1 and timestamp (32941163 273325056) Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK3 with disk number:2 and timestamp (32941163 273325056) Checking if the diskgroup is mounted Checking disk number:0 Checking disk number:1 Checking disk number:2 KFNDG-00600: file not found; arguments: [600] [Failed to initialize CSS context ] [] LEM-00031: Error encountered in lempgmh after calling lmserr. KFNDG-00600: file not found; arguments: [600] [Failed to initialize CSS context ] [] Terminating kgfd context 0x2b592002f0a0
As we can see it failed informing that CSS is not up. Then we tried renamedg command by starting cluster (ASM Diskgroup was mounted) but it too failed with error
Checking if the diskgroup is mounted Checking disk number:0 KFNDG-00405: file not found; arguments: [DATA]
Description of KFNDG clearly informs that diskgroup needs to be dismounted
KFNDG-00405: specified disk group string appears to be mounted Cause: Disk group was mounted. Action: Unmount the disk group and retry renamed
At this moment we thought that we cannot proceed and we would be required to re-install clusterware. Thinking over problem, I thought of moving the ocr and voting disk to different diskgroup and then renaming the old diskgroup.
Checked documentation and found out that there was command for replacing voting disk. In 11gR2 you can run this command while your cluster is up and running. So we created a new diskgroup DG_DATA01 and used following command
[oracle@prod-001]/dev/oracleasm/disks% crsctl replace votedisk +DG_DATA01 Failed to create voting files on disk group DG_DATA01. Change to configuration failed, but was successfully rolled back. CRS-4000: Command Replace failed, or completed with errors.
Oops. This failed. Checking ASM alert log gave the exact cause of this error. As you can see , it is asking for compatible.asm to be set to 11.2
NOTE: Creating voting files in diskgroup DG_DATA01 NOTE: Voting File refresh pending for group 2/0x88b6d50 (DG_DATA01) NOTE: Attempting voting file creation in diskgroup DG_DATA01 ERROR: Voting file allocation failed for group DG_DATA01 Errors in file /oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_4272.trc: ORA-15221: ASM operation requires compatible.asm of 11.2.0.0.0 or higher
Checking the compatability confirmed that it is set to 10.1
SQL> select name,COMPATIBILITY,DATABASE_COMPATIBILITY from V$ASM_DISKGROUP where name='DG_DATA01'; NAME COMPATIBILITY DATABASE_COMPATIBILITY -------- ---------------- -------------------------- DG_DATA01 10.1.0.0.0 10.1.0.0.0
You can use set attribute to change the compatability. e.g
SQL> alter diskgroup DG_DATA01 SET ATTRIBUTE 'compatible.asm'='11.2'; Diskgroup altered.
Retrying the command succeeded
[oracle@prod-001]/dev/oracleasm/disks% crsctl replace votedisk +DG_DATA01 Successful addition of voting disk 241b1e0a36344f7bbfaca4a576d514e9 . Successful deletion of voting disk 72e47e5e3afb4fe9bfb502b1b4340503. Successfully replaced voting disk group with +DG_DATA01. CRS-4266: Voting file(s) successfully replaced
We verified same using below command
[root@prod-001 ~]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 241b1e0a36344f7bbfaca4a576d514e9 (ORCL:DISK4) [DG_DATA01] Located 1 voting disk(s).
This is external redundancy diskgroup and oracle allows only one voting disk in external redundancy diskgroup.
Next was to change ocr location. This can be done using ocrconfig which needs to be run as root user. We tried using replace command but it didn’t work as it was the only OCR configured on cluster.
[root@prod-001 ~]# ocrconfig -replace +DATA -replacement +DG_DATA01 PROT-28: Cannot delete or replace the only configured Oracle Cluster Registry location
So we added additional OCR disk to new diskgroup and dropped old one
[root@prod-001 ~]# ocrconfig -add +DG_DATA01
[root@prod-001 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2424
Available space (kbytes) : 259696
ID : 579998313
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File Name : +DG_DATA01
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@prod-001 ~]# ocrconfig -delete +DATA
[root@prod-001 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2424
Available space (kbytes) : 259696
ID : 579998313
Device/File Name : +DG_DATA01
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
We then tried dropping diskgroup DATA
SQL> drop diskgroup data including contents; drop diskgroup data including contents * ERROR at line 1: ORA-15039: diskgroup not dropped ORA-15027: active use of diskgroup "DATA" precludes its dismount
It failed as ASM instance was using the parameter file which was kept on ASM diskgroup. So created a new pfile and then re-created spfile on new Diskgroup
SQL> create pfile from spfile; File created. SQL> create spfile='+DG_DATA01' from pfile; File created.
To drop the diskgroup , you will need to restart the cluster on both the nodes as ASM instance is still using the file. Run following commands on both the nodes
crsctl stop cluster crsctl start cluster
Verify that OCR location is pointing to the correct diksgroup on all nodes before starting cluster. We faced a issue in one more environment, where for some reason OCR location was not properly updated on second node. In that case we modified it manually (after confirming OCR location using ocrcheck)
[root@prod-002 ~]# cat /etc/oracle/ocr.loc #Device/file +DATA getting replaced by device +DG_DATA01 ocrconfig_loc=+DG_DATA01 local_only=false
You can verify the spfile and drop the old diskgroup now
SQL> sho parameter spfile NAME TYPE VALUE ------------------------------------ --------------------------------- ------------------------------ spfile string +DG_DATA01/prod-cluster/asmp arameterfile/registry.253.7290 69769 SQL> drop diskgroup data including contents; Diskgroup dropped.
In case you don’t wish to drop this diskgroup as it contains datafiles and would like to rename the diskgroup, you can refer to my earlier post 11gR2: Steps to Rename ASM Diskgroup with DB files
Recent Comments