ASM

Openfiler on Virtualbox and 12c Oracle Flex ASM

Last week Oracle released 12c database and Oracle blogosphere is bustling with lot of people posting details about new versions and setup.
I too decided to take plunge and setup my own RAC cluster. I had 4 machines with me but no shared storage 🙁
Long back I had done one installation using openfiler (Followed Jeff Hunter’s article on OTN) but in that case we installed openfiler software on base machine. Finally I decided to try installing it on Virtualbox.
I checked openfiler website and found out that they provide templates for VM which meant that installation on VM was supported. (Anyways this is my test setup 🙂 )

This was 64 bit machine, So I downloaded 64 bit rpm Virtualbox software.

When you try to install it, this will fail with following error

rpm -i VirtualBox-4.2-4.2.14_86644_el6-1.x86_64.rpm 
warning: VirtualBox-4.2-4.2.14_86644_el6-1.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 98ab5139: NOKEY
error: Failed dependencies:
	libSDL-1.2.so.0()(64bit) is needed by VirtualBox-4.2-4.2.14_86644_el6-1.x86_64

Solution : Install SDL package . You can either install SDL through yum or install same rpm using yum and it will find dependent SDL package and install it for you.

#yum install SDL
# rpm -i VirtualBox-4.2-4.2.14_86644_el6-1.x86_64.rpm 
warning: VirtualBox-4.2-4.2.14_86644_el6-1.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 98ab5139: NOKEY

Creating group 'vboxusers'. VM users must be member of that group!

No precompiled module for this kernel found -- trying to build one. Messages
emitted during module compilation will be logged to /var/log/vbox-install.log.

WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
Stopping VirtualBox kernel modules [  OK  ]
Recompiling VirtualBox kernel modules [  OK  ]
Starting VirtualBox kernel modules [  OK  ]

Add oracle user to vboxusers to allow oracle user to manage VM

usermod -G vboxusers oracle

Once done, you can download openfiler software from http://www.openfiler.com/community/download/. You can get direct link for 64 bit binary at Sourceforge.net

Let’s Build Virtual Machine for our setup.

1. Start virtualbox GUI by issuing virtualbox on command line
2. Click on New VM and choose OS (Linux) and Version (Red Hat 64 bit, change it based on your OS)
3. Allocate memory to Machine. I opted for 3G
4. Create a virtual drive of 30G and use .VDI as format. Even though usage is less then 8G, openfiler install fails if you create 8G disk.
5. Once done, click finish and click on settings.
6. Choose Network as eth0 and Bridged adapter. We are using single adapter here
7. Modify Boot order and remove floppy.Keep hard disk as first option
8. In System storage, You can add additional hard disk (Say 100G) which will be used to setup ASM devices

You can use screenshots from oracle-base 12c install article . Note that we are using single adapter and Bridged adapter.

8. When you start VM, it will ask you to choose start-up disk. Choose your openfiler.iso image
9. Press enter on boot screen and then click next

10. Choose install language
11. Next choose hard disk which will be used for installing software. Choose 30G disk which we allocated earlier. Uncheck the second disk

12. Next screen is network configuration. This is most important screen. I used static IP configuration , click edit for eth0 and put all required information i.e IP,subnet,gateway.Also add hostname and DNS information here.

13. Next select timezone and set root password. Once done, you will get success message and reboot will be done. If everything is successful, you can find your setup at

https://hostname:446/ (note its https and not http)

If this doesn’t work then look at your ip settings and ensure that your ip is pingable from outside. More troubleshooting for GUI can be done by restarting openfiler and httpd service. If it gives error, you can troubleshoot further

service openfiler restart
service httpd restart

You can configure openfiler ASM volumes by following Jeff Hunter article
In case you are on RHEL6, udev rules mentioned in above link will not work. Frits Hoogland article will be of help here (http://fritshoogland.wordpress.com/2012/07/23/using-udev-on-rhel-6-ol-6-to-change-disk-permissions-for-asm/)

You should be ready with ASM storage and can proceed with RAC install. There is not much difference in 12c install except that we have new feature called Oracle ASM Flex. I am documenting screenshots for same here

1. Choose Standard Cluster here. If you choose Flex cluster, it would force you to use GNS as option can’t be unchecked.

2. Choose Advanced Install

3. When choosing network interface, select ASM and private for private interface

4. On screen 10, choose Oracle Flex ASM

I did two Flex cluster setup with 3 node RAC and 2 node RAC and it seems to work at both places. Let’s see Flex cluster in action

You can verify if your ASM is enabled to use Flex mode using below command

[oracle@oradbdev02]~% asmcmd showclustermode
ASM cluster : Flex mode enabled

crsctl command can be used to set Flex mode later. Below command will show configuration

oracle@oradbdev02]~% srvctl config asm
ASM home: /home/oragrid
Password file: +VDISK/orapwASM
ASM listener: LISTENER
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM

[oracle@oradbdev02]~% srvctl status asm -detail
ASM is running on oradbdev02,oradbdev03,oradbdev04
ASM is enabled.

Lets reduce the ASM to run only on 2 nodes.This will stop ASM on one node

oracle@oradbdev02]~% srvctl modify asm -count 2

[oracle@oradbdev02]~% srvctl status asm -detail 
ASM is running on oradbdev02,oradbdev03
ASM is enabled.

There is no ASM on node oradbdev04 but db is still running

[oracle@oradbdev04]~% ps -ef|grep pmon
oracle    3949     1  0 10:27 ?        00:00:00 ora_pmon_snowy3
oracle   18728     1  0 07:59 ?        00:00:01 apx_pmon_+APX3

If you now try to start ASM on 3rd node, it will give error

[oracle@oradbdev02]~% srvctl config asm
ASM home: /home/oragrid
Password file: +VDISK/orapwASM
ASM listener: LISTENER
ASM instance count: 2
Cluster ASM listener: ASMNET1LSNR_ASM

[oracle@oradbdev02]~% srvctl start asm -n oradbdev04
PRCR-1013 : Failed to start resource ora.asm
PRCR-1064 : Failed to start resource ora.asm on node oradbdev04
CRS-2552: There are no available instances of resource 'ora.asm' to start.

Lets make it count back to 3 and ASM will start now

[oracle@oradbdev02]~% srvctl modify asm -count 3
[oracle@oradbdev02]~% srvctl start asm -n oradbdev04
[oracle@oradbdev02]~% srvctl status asm -detail     
ASM is running on oradbdev02,oradbdev03,oradbdev04
ASM is enabled.

Now need to explore some other new feature

2 July, 2013 by Amit Bansal 12c 0

Oracle Database Appliance Introduced

Oracle has introduced it new product: ODA (Oracle Database Appliance)

I was not able to join Oracle President Mark Hurd’s webcast where he supposed to announce a new Oracle product[It was at 1AM my time and I was sleeping], now viewing this webcast HERE

Kerry Osborne has shared information about this product, click link to see it, here you will see some good information about ODA.

Another new thing to talk about 🙂

22 September, 2011 by Saurabh Sood 11g 2

In-Place Upgrade 11gR2 RAC : 11.2.0.1 To 11.2.0.2

Sharing a post from my friend “Carthik” on 11gR2 RAC in-place upgrade.

Overview
========
Starting oracle 11gR2 the Oracle database and Clusterware upgrades are done via the “Out of place upgrade”. And is the easier way to perform your upgrade. However, the intent of this blog is to explain how an “In-Place upgrade” of a RAC database is done in 11gR2, the advantages, disadvantages, pre-requisites and pain points involved in this method which is the traditional method of upgrading an oracle database. And I have chosen to upgrade a 2 Node 11.2.0.1 RAC Database to 11.2.0.2 RAC Database.

Clearly, the advantage is that you save space, Instead of installing a new Oracle Home.

The greatest disadvantage is that you need to back up the oracle home and run detach oracle home commands. This could potentially damage your oracle binaries. However, when done carefully it should not cause a problem.

The pain points Include:
1. Backing up the Oracle home
2. Restoring the Oracle Home from backup in-case of installation failure.
3. Attach the restored home, and then bring up the instance.
Clearly, there is a lot of manual intervention, which is a major pain point.

The idea behind using an in-place upgrade is to save space. And this method of upgrade requires a significant amount time. The only time one should use this method is when you lack space on your server. However, you can use this method for your test and development environments to save space. Since this method was the traditional method of doing things, I thought it’s worth checking how it works in 11gR2.

NOTE: If you have an existing Oracle Clusterware installation, then you upgrade your existing cluster by performing an out-of-place upgrade. You cannot perform an in-place upgrade to the oracle Clusterware. However, you can perform an in-place upgrade for the database. This will be elaborated in this blog.
Prerequisites for Oracle 11gR2 11.2.0.2 installation is to install patch 9655006 to the 11.2.0.1 GI home before upgrading to 11.2.0.2 from 11.2.0.1. See Bug 9413827 on MOS. For additional details you can refer to the Metalink article Pre-requisite for 11.2.0.1 to 11.2.0.2 ASM Rolling Upgrade Metalink Note : 1274629.1. Additionally ,Refer to “How to Manually Apply A Grid Infrastructure PSU Without Opatch Auto” Metalink Note 1210964.1.

Performing an In-place upgrade of a RAC DB from 11.2.0.1 to 11.2.0.2

In-order to upgrade a RAC Database from 11.2.0.1 to 11.2.0.2 you need to upgrade

1. The grid infrastructure first.
2. Then, the Oracle Database.

Environment Setup Details used in this post:
— 2 Node RAC Red Hat Linux 5.5 with RACK Servers (R710).
— Applies to any storage.

Latest OPatch
It is recommended to we use the Latest Version of OPatch. Unzip the zip file and copy OPatch folder to $ORACLE_HOME And $GI_HOME by renaming
the earlier OPatch directory. You can refer to how to download and Install OPatch Metalink ID 274526.1.

Pre- Requisite Patch:

First, let’s discuss about the mandatory patches required before upgrading to 11.2.0.2. Patch 9655006 is required in order for the upgrade
to succeed, if not rootupgrade.sh will fail.
Patch 9655006 is downloadable from http://www.metalink.oracle.com/ For information on Bug 9655006, refer to Metalink article ID 9655006.8
Download the patch and unzip it to a stage directory (it can be any directory), once you unzip the downloaded patch, 2 folders will be created. In this example I have unzipped the patch to /patches directory.

Now, let’s see how to patch the $GI_HOME with Patch 9655006.

Before the patch is installed, we need to perform a pre-req check. Let’s see how to do this.
1. [root@rac1 patches]# su – oracle

2. [oracle@rac1 ~]$ cd /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/

3. [oracle@rac1 OPatch]$ ./opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /patches

Invoking OPatch 11.2.0.1.6
Oracle Interim Patch Installer version 11.2.0.1.6
Copyright (c) 2011, Oracle Corporation.  All rights reserved.
PREREQ session
Oracle Home       : /opt/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /opt/app/oraInventory
 from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.6
OUI version       : 11.2.0.1.0
Log file location : /opt/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2011-08-16_19-36-09PM.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.

Patching the $GI_HOME
1. Stop the Instance running on Node1
srvctl stop instance -d upgrade -i upgrade1

2. As root User run the opatch auto from the $GI_HOME
./opatch auto /patches

Note: The Opatch auto takes care of the patching of both the Grid infrastructure Home and the Oracle Home with the mandatory patch .
Once the patching is done on Node1, start the instance on Node1

3. Starting the Instance on Node1
srvctl start instance -d upgrade -i upgrade1

Repeat the process of pre-req and patching on Node2

Once the mandatory patch is applied, we can proceed with the upgrade of the grid infrastructure home.

Patches required:

The software/Patch can be downloaded from My Oracle support: patch 10098816. Select p10098816_112020_Linux-x86-64_3of7.zip
for grid infrastructure download. Once downloaded, unzip them.

Upgrading Grid Infrastructure:

Unzip the patches downloaded and invoke runInstaller from the unzipped grid folder. You will be taken to the welcome screen.

Choose Skip Software updates

Choose Upgrade Gird Infrastructure or Oracle ASM

Choose the Language

The Nodes present are selected by default, Click Next.

Leave the OS groups to Defaults

Choose the New Location where the Grid Infrastructure should be installed

The Pre-Requisite checks are performed, click next

The summary screen appears click next

Click on Install

Run rootupgrade.sh on both the nodes as specified in the screenshot

Upgrading the Database via In-place upgrade:

Patches Required:
The software/Patch can be downloaded from My Oracle support: patch 10098816.
Select p10098816_112020_Linux-x86-64_1of7.zip and p10098816_112020_Linux-x86-64_2of7.zip for database patch/software download.
Once downloaded, unzip them.

In-Place upgrades (Things to do before performing and In-place upgrade)

When performing an in-place upgrade, which uses the same Oracle home location, an error messages appears stating that the installer
detects Oracle Database software in the location that you specified.

Message: The installer has detected that the software location you have specified contains Oracle Database software release 11.2.0.1.
Oracle recommends that when upgrading to 11.2.0.2, you perform an out-of-place installation of the software into a new Oracle home and
then upgrade the database using the new software binaries.

Cause: The installer has detected that the software location you have specified contains Oracle Database software release 11.2.0.1.
Action: Either perform an in-place upgrade (Steps provided in this section), or perform an out-of-place upgrade

Performing an In-Place Upgrade for an Oracle RAC Database

To perform an in-place upgrade for Oracle RAC Database instances:
1. Back up the configuration data by backing up the following directories on all cluster nodes:
o ORACLE_HOME/dbs
o ORACLE_HOME/network/admin
o ORACLE_HOME/hostname_dbname
o ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_hostname_dbname

2. Run the following command on each of the nodes to detach the 11.2.0.1.0 Oracle RAC ORACLE_HOME:
$ORACLE_HOME/oui/bin/runInstaller -detachHome ORACLE_HOME=11.2.0.1.0 software location

3. Rename the 11.2.0.1.0 Oracle RAC ORACLE_HOME directory on all the nodes to a temporary name.

4. Install release 11.2.0.2 Software Only on all nodes:

From the unzipped folder, invoke the run Installer from the database folder

The welcome screen appears, uncheck the security updates and click next

Choose Skip Software Updates

Select Install Database Software Only and click next

Choose Oracle Real Application Clusters DB Installation and Select the Nodes and click next

Choose the Language and click next

Select Enterprise Edition and click next

Specify the location of the old home, and click next

Provide the Operating System groups and click next

Once the Pre-requisite checks are performed, click next

The summary screen appears, verify the settings and click next

The installation proceeds

Once the installation is done, run Root.sh on both the nodes as instructed and click ok.

Now, on all nodes, restore into the 11.2.0.2 ORACLE_HOME software location the backed up configuration data files
(from the backups you made of /dbs and network/admin), and also restore the following two directories:
/hostname_dbname and /oc4j/j2ee/OC4J_DBConsole_hostname_dbname. Specify the actual name for hostname_dbname.

Database Upgrade using DBUA:

Run DBUA from the 11.2.0.2 ORACLE_HOME/bin directory on the local node and select the 11.2.0.1.0 Oracle RAC database instance to
upgrade it to release 11.2.0.2.0.

The welcome screen appears once you invoke the DBUA, click next to proceed

DBUA Lists the databases that can be upgraded, select the one that you would like to upgrade

You can choose the Degree of parallelism and upgrading the time zone version and click next

Skip this screen by clicking next

The summary screen appears, click on finish for the upgrade to proceed.

The upgrade proceeds

NOTE: The only time one should use this method (in-place upgrade) is when you lack space on your server.
However, you can use this method for your test and development environments to save space.

Since this method was the traditional method of doing things,
Since this method is available, I thought it’s worth checking how it works in 11gR2.
During the entire upgrade process, I never ran into any issues, apart from the slightly higher downtime in
comparison to the out-of place upgrade. However, oracle doesn’t recommend this method. You can refer to the following metalink note 1291682.1.

8 September, 2011 by Saurabh Sood 11g Featured 12

Adding a reinstalled/reimaged node back to 11gR2 Cluster

There could be a scenario of node crash due to OS/hardware issues and a reinstall/reimage of the same. In such cases, just a normal node addition would not help since the OCR still contain the references of original node. We need to remove them first and then perform a node addition.

Have tried to document one such usecase.

Assumptions :
Cluster Hostnames : node1 , node2
VIP : node1-v , node2-v

– Voting disk and OCR are on ASM( ASMLIB is being used to manage the shared disks )
– After the OS reinstall, user equivalence has been set and all required packages have been installed along with setup of ASMLIB
– The crashed node was node2

STEPS
———-
1. Clearing the OCR entries for re-imaged host.

# crsctl delete node -n node2

To verify the success of above step, execute “olsnodes” on surviving node and the reimaged host shouldnot show up in list.

2. Remove the VIP information of reimaged host from OCR

Execute the following on existing node :
	/u01/grid/11.2/bin/srvctl remove vip -i node2-v -f

3. Clear the inventory for reimaged host for GI and DB Homes.

From the surviving node, execute :

/u01/grid/11.2/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/u01/grid/11.2 "CLUSTER_NODES=node1" CRS=TRUE -silent -local

Perform the similar for Database Home as well :

/u01/oracle/product/11.2/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/u01/oracle/product/11.2 CLUSTER_NODES=node1 -silent -local

4. Now starts the actual step of adding node. Run the Run the Cluster Verification Utility

./cluvfy  stage -pre nodeadd -n node2 -verbose

If possible, redirect the output of above to some file so that it can be reviewed and any issues reported can be rectified.

For this case, since the OCR and Voting disk resides on ASM and ASMLIB is in use, the most impacting errors were

ERROR:
PRVF-5449 : Check of Voting Disk location “ORCL:DISK6(ORCL:DISK6)” failed on the following nodes:
node2:No such file or directory

PRVF-5431 : Oracle Cluster Voting Disk configuration check failed

Will explain the impact of this error in the subsequent steps..

5. Run “addNode.sh” from existing node.

[oracle@node1] /u01/grid/11.2/oui/bin% ./addNode.sh -silent "CLUSTER_NEW_NODES={node2}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={node2-v}"
[oracle@node1] /u01/grid/11.2/oui/bin%

In my case, the above command came out without giving any messages. Actually the addNode.sh didnot run at all.

Cause : Since ASMLIB is in use, we had hit the issue discussed in MOS Note : 1267569.1
The error seen in step 4 helped in finding this.

Solution :

Set the following parameters and run addNode.sh again.

IGNORE_PREADDNODE_CHECKS=Y
export IGNORE_PREADDNODE_CHECKS

[oracle@node1] /u01/grid/11.2/oui/bin% ./addNode.sh -silent "CLUSTER_NEW_NODES={node2}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={node2-v}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 12143 MB

Performing tests to see whether nodes node2 are available
............................................................... 100% Done.

Cluster Node Addition Summary
Global Settings
Source: /u01/grid/11.2
New Nodes
Space Requirements
New Nodes
node2

Instantiating scripts for add node (Tuesday, December 21, 2010 3:35:16 AM PST)
. 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Tuesday, December 21, 2010 3:35:18 AM PST)
............................................................................................... 96% Done.
Home copied to new nodes

Saving inventory on nodes (Tuesday, December 21, 2010 3:37:57 AM PST)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each cluster node.
/u01/grid/11.2/root.sh # On nodes node2
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/grid/11.2 was successful.
Please check '/tmp/silentInstall.log' for more details.

6. Run root.sh on reimaged node to start up CRS stack.

This will completed Grid Infrastucture setup on the node.

7. Proceed to run addNode.sh for DB Home( on existing Node)

/u01/oracle/product/11.2/addNode.sh -silent "CLUSTER_NEW_NODES={node2}"

8. Once the DB Home addition is complete, use srvctl to check the status of registered DB and instances and add them if required.

2 May, 2011 by Abhishek 11g Featured 4

Renaming Diskgroup containing Voting Disk and OCR

Recently encountered a issue where we had to rename a diskgroup for 11.2 RAC containing OCR and Voting Disk.At this moment we had not created database .

I had done similar activity on 11.2 Single instance having database files (see this post) using renamedg. Therefore we tried using same approach to do this activity.

Pre-requisite for renamedg is that diskgroup should be dismounted. As this Diskgroup was storing OCR/Voting disk, it was not possible to dismount the diskgroup. So we stopped the cluster on both the nodes and tried running renamdg command

Old Diskgroup name – DATA
New Diskgroup name – DG_DATA01

[oracle@prod-001]~% renamedg phase=both dgname=DATA newdgname=DG_DATA01 config=/oracle/renamedg.out asm_diskstring='ORCL:DISK*' verbose=true keep_voting_files=true

Parsing parameters..

Parameters in effect:

 	 Old DG name       : DATA
	 New DG name          : DG_DATA01
	 Phases               :
	 	 Phase 1
	 	 Phase 2
	 Discovery str        : ORCL:DISK*
	 Keep_voting_files  : TRUE
	 Clean              : TRUE
	 Raw only           : TRUE
renamedg operation: phase=both dgname=DATA newdgname=DG_DATA01 config=/oracle/renamedg.out asm_diskstring=ORCL:DISK* verbose=true keep_voting_files=true
Executing phase 1
Discovering the group
Performing discovery with string:ORCL:DISK*
Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK1 with disk number:0 and timestamp (32941163 273325056)
Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK2 with disk number:1 and timestamp (32941163 273325056)
Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK3 with disk number:2 and timestamp (32941163 273325056)
Checking for hearbeat...
Re-discovering the group
Performing discovery with string:ORCL:DISK*
Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK1 with disk number:0 and timestamp (32941163 273325056)
Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK2 with disk number:1 and timestamp (32941163 273325056)
Identified disk ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so:ORCL:DISK3 with disk number:2 and timestamp (32941163 273325056)
Checking if the diskgroup is mounted
Checking disk number:0
Checking disk number:1
Checking disk number:2
KFNDG-00600: file not found; arguments: [600] [Failed to initialize CSS context
] []
LEM-00031: Error encountered in lempgmh after calling lmserr.
KFNDG-00600: file not found; arguments: [600] [Failed to initialize CSS context
] []

Terminating kgfd context 0x2b592002f0a0

As we can see it failed informing that CSS is not up. Then we tried renamedg command by starting cluster (ASM Diskgroup was mounted) but it too failed with error

Checking if the diskgroup is mounted
Checking disk number:0
KFNDG-00405: file not found; arguments: [DATA]

Description of KFNDG clearly informs that diskgroup needs to be dismounted

KFNDG-00405: specified disk group string appears to be mounted
Cause: Disk group was mounted.
Action: Unmount the disk group and retry renamed

At this moment we thought that we cannot proceed and we would be required to re-install clusterware. Thinking over problem, I thought of moving the ocr and voting disk to different diskgroup and then renaming the old diskgroup.

Checked documentation and found out that there was command for replacing voting disk. In 11gR2 you can run this command while your cluster is up and running. So we created a new diskgroup DG_DATA01 and used following command

[oracle@prod-001]/dev/oracleasm/disks% crsctl replace votedisk +DG_DATA01
Failed to create voting files on disk group DG_DATA01.
Change to configuration failed, but was successfully rolled back.
CRS-4000: Command Replace failed, or completed with errors.

Oops. This failed. Checking ASM alert log gave the exact cause of this error. As you can see , it is asking for compatible.asm to be set to 11.2

NOTE: Creating voting files in diskgroup DG_DATA01
NOTE: Voting File refresh pending for group 2/0x88b6d50 (DG_DATA01)
NOTE: Attempting voting file creation in diskgroup DG_DATA01
ERROR: Voting file allocation failed for group DG_DATA01
Errors in file /oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_4272.trc:
ORA-15221: ASM operation requires compatible.asm of 11.2.0.0.0 or higher

Checking the compatability confirmed that it is set to 10.1

SQL> select name,COMPATIBILITY,DATABASE_COMPATIBILITY from V$ASM_DISKGROUP where name='DG_DATA01';

NAME       COMPATIBILITY      DATABASE_COMPATIBILITY
--------  ----------------   --------------------------
DG_DATA01  10.1.0.0.0        10.1.0.0.0

You can use set attribute to change the compatability. e.g

SQL> alter diskgroup DG_DATA01 SET ATTRIBUTE 'compatible.asm'='11.2';

Diskgroup altered.

Retrying the command succeeded

[oracle@prod-001]/dev/oracleasm/disks% crsctl replace votedisk +DG_DATA01
Successful addition of voting disk 241b1e0a36344f7bbfaca4a576d514e9                                                                                                                                                                                                                                .
Successful deletion of voting disk 72e47e5e3afb4fe9bfb502b1b4340503.
Successfully replaced voting disk group with +DG_DATA01.
CRS-4266: Voting file(s) successfully replaced

We verified same using below command

[root@prod-001 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   241b1e0a36344f7bbfaca4a576d514e9 (ORCL:DISK4) [DG_DATA01]
Located 1 voting disk(s).

This is external redundancy diskgroup and oracle allows only one voting disk in external redundancy diskgroup.

Next was to change ocr location. This can be done using ocrconfig which needs to be run as root user. We tried using replace command but it didn’t work as it was the only OCR configured on cluster.

[root@prod-001 ~]# ocrconfig -replace +DATA -replacement +DG_DATA01
PROT-28: Cannot delete or replace the only configured Oracle Cluster Registry location

So we added additional OCR disk to new diskgroup and dropped old one

[root@prod-001 ~]# ocrconfig -add +DG_DATA01
[root@prod-001 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
	 Version                  :          3
	 Total space (kbytes)     :     262120
	 Used space (kbytes)      :       2424
	 Available space (kbytes) :     259696
	 ID                       :  579998313
	 Device/File Name         :      +DATA
                                    Device/File integrity check succeeded
	 Device/File Name         : +DG_DATA01
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

	 Cluster registry integrity check succeeded

	 Logical corruption check succeeded

[root@prod-001 ~]# ocrconfig -delete +DATA
[root@prod-001 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
	 Version                  :          3
	 Total space (kbytes)     :     262120
	 Used space (kbytes)      :       2424
	 Available space (kbytes) :     259696
	 ID                       :  579998313
	 Device/File Name         : +DG_DATA01
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

	 Cluster registry integrity check succeeded

	 Logical corruption check succeeded

We then tried dropping diskgroup DATA

SQL> drop diskgroup data including contents;
drop diskgroup data including contents
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15027: active use of diskgroup "DATA" precludes its dismount

It failed as ASM instance was using the parameter file which was kept on ASM diskgroup. So created a new pfile and then re-created spfile on new Diskgroup

SQL> create pfile from spfile;
File created.

SQL> create spfile='+DG_DATA01' from pfile;

File created.

To drop the diskgroup , you will need to restart the cluster on both the nodes as ASM instance is still using the file. Run following commands on both the nodes

crsctl stop cluster
crsctl start cluster

Verify that OCR location is pointing to the correct diksgroup on all nodes before starting cluster. We faced a issue in one more environment, where for some reason OCR location was not properly updated on second node. In that case we modified it manually (after confirming OCR location using ocrcheck)

[root@prod-002 ~]# cat /etc/oracle/ocr.loc
#Device/file +DATA getting replaced by device +DG_DATA01
ocrconfig_loc=+DG_DATA01
local_only=false

You can verify the spfile and drop the old diskgroup now

SQL> sho parameter spfile

NAME				     TYPE			       VALUE
------------------------------------ --------------------------------- ------------------------------
spfile				     string			       +DG_DATA01/prod-cluster/asmp
								       arameterfile/registry.253.7290
								       69769
SQL> drop diskgroup data including contents;

Diskgroup dropped.

In case you don’t wish to drop this diskgroup as it contains datafiles and would like to rename the diskgroup, you can refer to my earlier post 11gR2: Steps to Rename ASM Diskgroup with DB files

9 September, 2010 by Amit Bansal 11g Featured 7

How To Discover Disk Name When LUN Number is Known (OEL5)

I am creating 11gR2 RAC setup for one of my client. Following Oracle documentation for storage, I opted for Oracle ASM and asked storage team for new physical devices.
Storage admin thus provided me with a set of LUNs instead of actual physical device names like /dev/sdcxxx.

Now the major task is to get the actual device name associated with LUNs.
In OEL4 this is easy to get by issuing

# iscsi-ls -l
This command will give output like:

<span style="font-size: medium;"><span style="font-family: times new roman,times;">DEVICE DETAILS:
---------------
LUN ID : 0
Vendor: DELL     Model: MD3000i          Rev: 0670
Type:   Direct-Access                    ANSI SCSI revision: 05
page83 type3: 6001e4f0003fa9970000083c48573c94
page80: 383438303045500a
Device: /dev/sdh
LUN ID : 1
Vendor: DELL     Model: MD3000i          Rev: 0670
Type:   Direct-Access                    ANSI SCSI revision: 05
page83 type3: 6001e4f0003fbc5b000008dd486dae67
page80: 383438303045500a
Device: /dev/sdl
LUN ID : 2
Vendor: DELL     Model: MD3000i          Rev: 0670
Type:   Direct-Access                    ANSI SCSI revision: 05
page83 type3: 6001e4f0003fbc5b0000082048574033
page80: 383438303045500a
Device: /dev/sdp</span></span>

But this command is obsolete in OEL5. The replacement for iscsi-ls in OEL5 is iscsiadm.

In OEL5, there are two ways to get the device name associated with LUN:

1. Use iscsiadm -m host -P 4

# iscsiadm -m host -P 4

It will show the output as:

<span style="font-size: medium;"><span style="font-family: times new roman,times;">************************
Attached SCSI devices:
************************
scsi6 Channel 00 Id 0 Lun: 0
Attached scsi disk sdk          State: running
scsi6 Channel 00 Id 0 Lun: 1
Attached scsi disk sdm          State: running
scsi6 Channel 00 Id 0 Lun: 10
Attached scsi disk sdah         State: running
scsi6 Channel 00 Id 0 Lun: 11
Attached scsi disk sdai         State: running
scsi6 Channel 00 Id 0 Lun: 13
Attached scsi disk sdap         State: running
scsi6 Channel 00 Id 0 Lun: 14
Attached scsi disk sdaq         State: running</span></span>

2. Install the OEL5 package: lsscsi-0.17-3.el5 (or similar version).

# lsscsi -l

<span style="font-size: medium;"><span style="font-family: times new roman,times;">It will show the output as:</span></span>

<span style="font-size: medium;"><span style="font-family: times new roman,times;">[6:0:0:22]   disk    DELL     MD3000i          0670  /dev/sdbp
state=running queue_depth=32 scsi_level=6 type=0 device_blocked=0 timeout=60
[6:0:0:24]   disk    DELL     MD3000i          0670  /dev/sdbq
state=running queue_depth=32 scsi_level=6 type=0 device_blocked=0 timeout=30
[6:0:0:31]   disk    DELL     Universal Xport  0670  /dev/sdbx
state=running queue_depth=32 scsi_level=6 type=0 device_blocked=0 timeout=60
[6:0:0:32]   disk    DELL     MD3000i          0670  /dev/sdby
state=running queue_depth=32 scsi_level=6 type=0 device_blocked=0 timeout=30
[6:0:0:33]   disk    DELL     MD3000i          0670  /dev/sdcg
state=running queue_depth=32 scsi_level=6 type=0 device_blocked=0 timeout=60
[6:0:0:71]   disk    DELL     MD3000i          0670  /dev/sdch
state=running queue_depth=32 scsi_level=6 type=0 device_blocked=0 timeout=60
[6:0:0:72]   disk    DELL     MD3000i          0670  /dev/sdci
state=running queue_depth=32 scsi_level=6 type=0 device_blocked=0 timeout=30
[6:0:0:73]   disk    DELL     MD3000i          0670  /dev/sdck
state=running queue_depth=32 scsi_level=6 type=0 device_blocked=0 timeout=60</span></span>

<span style="font-size: medium;"><span style="font-family: times new roman,times;">Here we can see the LUN numbers in begning:</span></span>

[6:0:0:73] i.e 73 is the LUN number here.

[6:0:0:72] i.e 72 is the LUN number here.

Finally I created the disks with correct device names. 🙂

Cheers!!!

Saurabh Sood

1 February, 2010 by Saurabh Sood 10g 11g 2

ASM

Openfiler on Virtualbox and 12c Oracle Flex ASM

Oracle Database Appliance Introduced

In-Place Upgrade 11gR2 RAC : 11.2.0.1 To 11.2.0.2

Adding a reinstalled/reimaged node back to 11gR2 Cluster

Renaming Diskgroup containing Voting Disk and OCR

How To Discover Disk Name When LUN Number is Known (OEL5)

Disclaimer!!

Recent Comments

Trending

Subscribe to Blog via Email