ASM

Creating ASM devices on AIX

I thought of sharing few tips on creating ASM devices on AIX which I will be helpful to Oracle DBA’s. Suppose SysAdmin gives you list of Serial numbers for LUN instead of device Name

pcmpath query device
DEV#:  33  DEVICE NAME: hdisk33  TYPE: 2107900  ALGORITHM:  Load Balance
SERIAL: 75DM011<span style="color: #ff0000;"><strong>1101</strong></span>
===========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0           fscsi0/path0          CLOSE   NORMAL          9          0
    1           fscsi1/path1          CLOSE   NORMAL          8          0

In case there are lot many disks, then it could be a tiring task of running above command and finding each device. You can use below code which will list name of  devices and size (In MB) of disk.

for i in 1000 1100    1018    1118    1030    1130    104C    114C    1068    1168    1080    1180
do
j=`pcmpath query device|grep -p $i"$"|grep DEVICE|awk -F ":" '{print }'|awk '{print }`
k=`bootinfo -s $j`
echo $i $j $k
done

This would return following output

1000 hdisk4 65536
1100 hdisk10 65536
1018 hdisk5 65536
1118 hdisk11 65536
1030 hdisk6 65536
1130 hdisk12 65536
104C hdisk7 65536
114C hdisk13 65536
1068 hdisk8 65536
1168 hdisk14 65536
1080 hdisk9 65536
1180 hdisk15 65536

Now if you need to create new device name, you need to use mknod command and pass on major and minor numbers. Following code can be used to perform same

#export m=0
 # for i in hdisk4  hdisk10 hdisk5  hdisk11 hdisk6  hdisk12 hdisk7  hdisk13 hdisk8  hdisk14 hdisk9  hdisk15
 do
 j=`ls -la /dev/$i |awk '{print }'|awk -F "," '{print }'`
  k=`ls -la /dev/$i |awk '{print }'`
 m=`expr $m + 1` ;echo "mknod /dev/asm_disk"$m "c "$j $k
 done

 mknod /dev/asm_disk1 c 21 4
 mknod /dev/asm_disk2 c 21 12
 mknod /dev/asm_disk3 c 21 13
 mknod /dev/asm_disk4 c 21 15
 mknod /dev/asm_disk5 c 21 5
 mknod /dev/asm_disk6 c 21 6
 mknod /dev/asm_disk7 c 21 8
 mknod /dev/asm_disk8 c 21 7
 mknod /dev/asm_disk9 c 21 14
 mknod /dev/asm_disk10 c 21 10
 mknod /dev/asm_disk11 c 21 9
 mknod /dev/asm_disk12 c 21 11

Now you can change the ownership to oracle:dba and permission to 660. I have 12 disks , so using list of 12 variables. In case you have more disks , then you can add more variables

# for i in 1 2 3 4 5 6 7 8 9 10 11 12
do
chown oracle:dba /dev/asm_disk$i
chmod 660 /dev/asm_disk$i
done

crw-rw----    1 oracle   dba          21, 11 Jan 28 17:10 /dev/asm_disk12
crw-rw----    1 oracle   dba          21,  9 Jan 28 17:10 /dev/asm_disk11
crw-rw----    1 oracle   dba          21, 10 Jan 28 17:10 /dev/asm_disk10
crw-rw----    1 oracle   dba          21, 14 Jan 28 17:04 /dev/asm_disk9
crw-rw----    1 oracle   dba          21,  7 Jan 28 17:04 /dev/asm_disk8
crw-rw----    1 oracle   dba          21,  8 Jan 28 17:04 /dev/asm_disk7
crw-rw----    1 oracle   dba          21,  6 Jan 28 17:04 /dev/asm_disk6
crw-rw----    1 oracle   dba          21,  5 Jan 28 17:04 /dev/asm_disk5
crw-rw----    1 oracle   dba          21, 15 Jan 28 17:04 /dev/asm_disk4
crw-rw----    1 oracle   dba          21, 13 Jan 28 17:04 /dev/asm_disk3
crw-rw----    1 oracle   dba          21, 12 Jan 28 17:04 /dev/asm_disk2
crw-rw----    1 oracle   dba          21,  4 Jan 28 17:04 /dev/asm_disk1

In case you need to use same logic for creating OCR and Voting disks on RAC system, replace /dev/asm with /dev/ocr or /dev/voting . I hope this would save some time and also prevent errors 🙂

In case you have disks in ordered number,say 53 to 62 then you can also use for loop as below.

#bash
bash-3.00#
#export m=0
#for ((i=53;i<=62;i++))
do
 j=`ls -la /dev/hdisk$i |awk '{print }'|awk -F "," '{print }'`
  k=`ls -la /dev/hdisk$i |awk '{print }'`
   m=`expr $m + 1` ;echo "mknod /dev/asm_disk"$m "c "$j $k
 done
#for ((i=1;i<=10;i++))
do
chown oracle:dba /dev/asm_disk$i
chmod 660 /dev/asm_disk$i
done

I would suggest anyone using the scripts to first check in a test environment.

Creating Oracle Extended RAC on Oracle VM

Yesterday, I found one very useful article at OTN “Creating Oracle Extended RAC” on completely virtual environment using Oracle VM. As Virtualization is becoming popular day by day and is very cost effective, one must know how to use this to simulate actual environments. Click  here for details on Oracle Extended RAC on Oracle VM.

Question:Sizing ASM LUN

We received following question from Mr Orlando through our Contact Page

How about a discussion on how to size our LUNS to use with ASM. For instance we have a Dell MD3000 with 44x300GB HDDs (protected by RAID-10) and we’re planning to create LUNs of 2TB on the Storage, and deliver them to Windows 2003. for ASM to use what is common around the world? Just pass the 2TB Luns to Windows or maybe create several 500GB and give them to ASM. This is a RAC10g install, Oracle 10g Standard Edition.

Please find below our response to the question. We would also like to hear from others on their experiences and any other suggestions/opinions they may gave.

Our Response

As per 10gR2 ASM Best Practices document (available on ASM OTN Homepage), we can add Disk having size up to 2^32 Mb as ASM Disk (Minimum being 4 Mb). Therefore technically 2 Tb Disk/LUN can be added to the ASM Diskgroup but we would not recommend so.

Only advantage of having 2 Tb LUN would be reduced ASM Disks resulting in easier management. But there are some pitfalls too. We believe most of the problems will be seen during Disk Rebalance.

Suppose we have 3 TB of data on Two ASM Disks of 2 Tb each. Now we need to add new disk of 2Tb.In this case 1 Tb of data movement will happen and it will impact DB performance . Rebalance operation can also happen in case we are using Normal/High Redundancy and Disk failure occurs (due to Hardware problem or issues like header or block corruption), then failure group would be in Hung State. This needs to be corrected by adding a new Disk .

This would again require large amount of data movement (1.5 Tb in this example) and will take longer time to finish.

Lun size of 300-500 Gb should be fine for VLDB configuration.  Having separate diskgroup for DATA and FRA (Flash Recover Area) is also recommended approach. One more thing which can be tried out in case of multi-terabyte database is to change the Allocation Unit size for ASM Diskgroup.Each Allocation Unit consumes some amount of memory in ASM SGA for storing the metadata information. Very Large Databases (VLDB) which can have sizes in TeraBytes will have too many AU’s allocated and also memory requirement for ASM instance will also increase. You can read more about this in our previous post Allocation Unit and Extents In ASM

Raw Devices to be deprecated in 12G

I was browsing through the OraNA.info posts and found an interesting post from Steve Karam which referred to a metalink Note:578455.1 – Announcement of De-Support of RAW devices in Release 12G

If you go through the note, it mentions desupport of Raw devices from Oracle Database 12G. Article also lists out possibility of using ASM,OCFS as few of the alternative storage mediums for keeping OCR and Voting Disks (Used in Oracle Real Application Clusters(RAC)) .

OCFS2 already supports the storage of OCR and Voting Disk. But note also talks about ASM will be supporting the files. Hmmm… If this has to be true, lot of changes will be required in the architecture. Currently ASM instance starts after CSS (and other clusterware services in RAC) service has been started . But this change will mean that ASM has to start before these processes. Currently if you try to start the ASM instance with CSS service down, you get following error

[oracle@blrraclnx2 ~]$ export ORACLE_SID=+ASM1
[oracle@blrraclnx2 ~]$ sqlplus 

SQL*Plus: Release 10.2.0.1.0 - Production on Sun Aug 10 08:16:09 2008

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Enter user-name: / as sysdba
Connected to an idle instance.

SQL> startup
ORA-29701: unable to connect to Cluster Manager
SQL> exit
Disconnected
[oracle@blrraclnx2 ~]$ oerr ora 29701
29701, 00000, "unable to connect to Cluster Manager"
// *Cause: Connect to CM failed or timed out.
// *Action: Verify that the CM was started.  If the CM was not started,
//          start it and then retry the database startup. If the CM died
//          or is not responding, check the Oracle and CM trace files for
//          errors.

Apart from changing the architecture, it will also involve lot of effort from Oracle DBA’s to unlearn and learn new concepts 🙂 At the same time, it will help you start the ASM Instance even though CSS is not up!! (I know many people will be having a sigh of relief after reading the last line) Or is there something else in store for us! There are also lot of RAC and ASM features expected in 11gR2. So let’s wait and watch..

Automatic Storage Management (ASM) FAQ

We have tried to consolidate list of Frequently Asked Questions (FAQ) on ASM. Please feel free to ask any other question or contribute to this FAQ by using comment section

Frequently Asked Question on ASM

1) What is ASM?

Automatic storage Management (ASM) is a new type of filesystem which was introduced with Oracle 10g. ASM is recommended filesystem for RAC and Single instance ASM for storing database files. This provides direct I/O to the file and performance is comparable with that provided by RAW Devices. Oracle creates a separate instance for this purpose.

2) How do we identify if we are connected to Normal Instance or ASM instance?

Issue following command to identify this

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">
SQL&gt; show parameter instance_type</span>

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
instance_type string asm</span>

In case you are connected to ASM instance, it will display value as asm, otherwise it will display value as RDBMS.

3) What are Diskgroups and Failuregroups?

Diskgroup is a terminology used for logical structure which holds the database files. Each Diskgroup consists of Disks/Raw devices where the files are actually stored. Any ASM file is completely contained within a single disk group. However, a disk group might contain files belonging to several databases and a single database can use files from multiple disk groups.

Failuregroups are used when using Normal/High Redundancy. They contain the mirrored ASM extents and must be containing different disks and preferably on separate disk controller.

4)Can ASM be used as replacement for RAID?

ASM is supposed to stripe the data and also mirror the data (if Using Normal, High Redundancy). So this can be used as a alternative for RAID 0+1 solutions

5) How does ASM provides Redundancy?

When you create a disk group, you specify an ASM disk group type based on one of the following three redundancy levels:

  • Normal for 2-way mirroring – When ASM allocates an extent for a normal redundancy file, ASM allocates a primary copy and a secondary copy. ASM chooses the disk on which to store the secondary copy in a different failure group other than the primary copy.
  • High for 3-way mirroring. In this case the extent is mirrored across 3 disks.
  • External to not use ASM mirroring. This is used if you are using Third party Redundancy mechanism like RAID, Storage arrays.

6) Can we change the Redundancy for Diskgroup after its creation.

No, we cannot modify the redundancy for Diskgroup once it has been created. To alter it we will be required to create a new Diskgroup and move the files to it. This can also be done by restoring full backup on the new Diskgroup. Following metalink note describes the steps

Note.438580.1 – How To Move The Database To Different Diskgroup (Change Diskgroup Redundancy)

7) I am unable to open the ASM instance. What is the reason?

ASM instance does not have open stage. It has got only two options

  • Nomount- This starts the ASM instance
  • Mount- At this stage, Diskgroup defined in ASM_DISKGROUPS parameter are mounted

When you try to open the ASM instance , you get following error

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">SQL&gt; alter database open;
alter database open
*
ERROR at line 1:
ORA-15000: command disallowed by current instance type</span>

8)Can ASM instance and database (rdbms) be on different servers?

ASM instance and Database (rdbms) have to be present on same server. Otherwise it will not work.


9)
Can we see the files stored in the ASM instance using standard unix commands.

No, you cannot see the files using standard unix commands like ls. You need to use utility called asmcmd to do this. This is present in 10.2 and above.e.g
/home/oracle>asmcmd
Asmcmd>

You can use help command to see the options.

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">Note: - You can use asmcmd for 10.1 database also. For this you can copy the </span><span style="font-size: small; font-family: arial,helvetica,sans-serif;">asmcmdcore and asmcmd file from 10.2 ORACLE_HOME to 10.1 ORACLE_HOME. Else you can download the files from Metalink </span><span style="font-size: small; font-family: arial,helvetica,sans-serif;"><strong>Note:332853.1</strong></span>

10) What kind of files can be stored on ASM Diskgroup.

You can store the following file types in ASM disk groups:

  • Control files
  • Datafiles, temporary datafiles, and datafile copies
  • SPFILEs
  • Online redo logs, archive logs, and Flashback logs
  • RMAN backups
  • Disaster recovery configurations
  • Change tracking bitmaps
  • Data Pump dumpsets

Note: Oracle executables and ASCII files, such as alert logs and trace files, cannot be stored in ASM disk groups.

11)Can we use ASM for storing Voting Disk/OCR in a RAC instance?

No. You cannot use ASM for storing the voting disk and OCR. It is due to the fact that Clusterware starts before ASM instance and it should be able to access these files which is not possible if you are storing it on ASM. You will have to use raw devices or OCFS or any other shared storage.

12)Does ASM instance automatically rebalances and takes care of hot spots?

No. This is a myth and ASM does not do it. It will initiate automatic rebalance only when a new disk is added to Diskgroup or we drop a disk from existing Diskgroup.

13) What is ASMLIB?


ASMLIB is the support library for the ASM. ASMLIB allows an Oracle database using ASM more efficient and capable access to diskgroups. The purpose of ASMLIB, is to provide an alternative interface to identify and access block devices. Additionally, the ASMLIB API enables storage and operating system vendors to supply extended storage-related features. You can read more about ASMLIB in

Note: 359266.1 – FAQ ASMLIB CONFIGURE, VERIFY, TROUBLESHOOT

ASMLIB for linux can be downloaded from following link

http://www.oracle.com/technology/tech/linux/asmlib/index.html

14) What is SYSASM role?
Starting from Oracle 11g, SYSASM role can be used to administer the ASM instances. You can continue using SYSDBA role to connect to ASM but it will generate following warning messages at time of startup/shutdown, create Diskgroup/add disk ,etc

<span style="font-size: small; font-family: arial,helvetica,sans-serif;">Alert entry
WARNING: Deprecated privilege SYSDBA for command 'STARTUP'</span>

15) How can we copy the files from/to ASM?

You can use RMAN or DBMS_FILE_TRANSFER.COPY_FILE procedure to copy the files to/from ASM from/to Filesystem. Starting from Oracle 11g, you can use cp command in asmcmd to perform the same between ASM Diskgroups and also to OS Filesystem.

16) Can we use BCV to clone the ASM Diskgroup on same host?

Diskgroup Cloning is not supported on the same host using BCV. You have no other option to use except RMAN DUPLICATE. You can find more detail on BCV and ASM in below whitepaper

http://www.oracle.com/technology/products/database/asm/pdf/asm-on-emc-5_3.pdf

17) Can we edit the ASM Disk header to change the Diskgroup Name?

No. This cannot be done. Please find details in

http://askdba.org/weblog/?p=128

18) Whats is Kfed?

kfed is a utility which can be used to view the ASM Disk information. Syntax for using it is

kfed read devicename

Read more about kfed here.

19) Can we use block devices for ASM Disks?

Yes. Starting from Oracle Database 10.2 block devices can be used directly for ASM Disks in Linux. This is not true for other Unix based systems where block devices are not supported yet.

Along with this it is recommended to use a Device mapping functionality so that disk mapping is preserved after disk failure. This is important when you have devices as /dev/sda,/dev/sdb,/dev/sdc and due to some reason the devices are not detected at next reboot (say /dev/sdb), the system will map the incorrect device (i.e /dev/sdc will be marked as /dev/sdb). You can use following methods for preserving disk names

-udev – the role of udev is to provide device persistency and naming consistency.This is especially important for the Oracle Cluster Registry (OCR) and Voting disks required by Oracle Clusterware.

ASMLIB – ASMLIB will provide device management specifically for ASM disk devices.

Refer to following Whitepaper for more details

http://www.oracle.com/technology/products/database/asm/pdf/device-mapper-udev-crs-asm%20rh4.pdf

20)Is it mandatory to use disks of same size and characteristics for Diskgroups?

No, it is not mandatory to use the disks of same size and characteristics for Diskgroups though it is a Recommended Practice.

Same size disk for Failuregroups in Normal/High redundancy will prevent issues like ORA-15041 as the file extents needs to be mirrored across the disks. Also as Oracle distributes data based on capacity, so larger disk will have more data stored in it and which will result in higher I/O to disk and eventually can lead to sub-optimal performance.

Moreover having disks of different characteristic like varying disk speed can impact the performance.

When managing disks with different size and performance capabilities, best practice is to group them into disk groups according to their characteristics. So you can use higher speed disks for your database files while other disks can be part of Diskgroup used for Flash Recovery Area.

21)Do we need to install ASM and Oracle Database Software in different ORACLE_HOME?

No. Again installing ASM and Oracle Database Software in different ORACLE_HOME is not mandatory but a best practice. This is useful in cases when we need to have multiple databases using same ASM instance and you need to patch only one of them. E.g You need to apply a CBO patch to one of 10.2 database while your other 10.1 database using different installation does not require it. In this case having a ASM_HOME separate from 10.2 ORACLE_HOME will allow your 10.1 database to keep running. Thus this approach is useful for High Availability.

22)What is the maximum size of Disk supported by ASM?

ASM supports disks upto 2Tb, so you need to ensure that lun size should be less then 2Tb. 10.2.0.4 and 11g database will give error if you try to create a diskgroup with ASM disks having disk size >2Tb.

23)I have created Oracle database using DBCA and having a different home for ASM and Oracle Database. I see that listener is running from ASM_HOME. Is it correct?

This is fine. When using different home for ASM, you need to run the listener from ASM_HOME instead of ORACLE_HOME.

Additional Links for Further Reading

http://www.oracle.com/technology/products/database/asm/index.html http://download.oracle.com/docs/cd/B28359_01/server.111/b31107/toc.htm

http://www.oracle.com/technology/products/database/asm/pdf/device-mapper-udev-crs-asm%20rh4.pdf
Note:359266.1 – FAQ ASMLIB CONFIGURE,VERIFY, TROUBLESHOOT
Note 266028.1 – ASM Using Files Instead of Real Devices on Linux
Note.438580.1 – How To Move The Database To Different Diskgroup (Change Diskgroup Redundancy)

11G:SYSASM role

Starting from Oracle 11g ASM, new privilege has been created for Administering Oracle ASM instances. This is called SYSASM role.

You can continue using SYSDBA role to connect to ASM but it will generate following warning messages at time of startup/shutdown, create Diskgroup/add disk, etc.

<strong>$ adrci</strong>

ADRCI: Release 11.1.0.6.0 - Beta on Tue Jul 1 15:43:57 2008
Copyright (c) 1982, 2007, Oracle.  All rights reserved.

ADR base = "/u03/app/oracle"
adrci&gt; help
adrci&gt; set homes
DIA-48431: Must specify at least one ADR home path

<strong>adrci&gt; show homes</strong>
ADR Homes:
diag/rdbms/testdb11/TESTDB11
diag/asm/+asm/+ASM
diag/clients/user_oracle/host_2900411789_11
diag/tnslsnr/testzone/listener
<strong>adrci&gt; set home diag/asm/+asm/+ASM
adrci&gt; show alert</strong>

ADR Home = /u03/app/oracle/diag/asm/+asm/+ASM:
******************************************
Starting ORACLE instance (normal)
<strong>WARNING: Deprecated privilege SYSDBA for command 'STARTUP'</strong>
2008-06-25 13:13:09.287000 +05:30

Please note that SYSASM privilege cannot be used to start the RDBMS instance. This way it ensures separation of database administration privilege from the ASM storage administration privilege. Trying to do so results in ORA – 1031

$ sqlplus
SQL*Plus: Release 11.1.0.6.0 - Production on Tue Jul 1 16:09:54 2008
Copyright (c) 1982, 2007, Oracle.  All rights reserved.
Enter user-name: / as sysasm
Connected to an idle instance.

SQL&gt; startup pfile=initest.ora
<strong>ORA-01031: insufficient privileges</strong>

As per documentation, the privilege to administer an ASM instance with SYSDBA will be removed in future versions. Anyways till now DBA’s can administer the ASM instance 😉