AIX5L

Installing 10.2.0.5 EM using software-only on 10.2.0.4/11g DB (GUI)

In this article , I will discuss how to install 10.2.0.5 EM Grid using 10.2.0.4 database. This method can also be used with 11g database.When you carry out 10.2.0.1 EM Grid control installation, it creates a 10.1.0.4 Database by default which is no longer supported and  requires one to upgrade to supported release i.e 10.2. In case you choose EM grid installation with existing database, then DB version need’s to be <=10.2.0.3 which you would again like to upgrade to so as to easily receive bug fixes and be on latest supported release. So if you want to use 10.2.0.4 DB or 11.1 database, you will not be able to install using existing database directly but would have to follow a upgrade path. To overcome such situation, starting 10.2.0.4 EM Grid Control we now have Software-Only install method .Using the ‘Installing Software-Only and Configuring Later’ installation method, you can install only the software of the base release and then configure it later by applying the latest patch set.

Note that 10.2.0.4 /10.2.0.5 are patchset’s and need a 10.2.0.1 base installation to work.

In case you wish to carry out the installation in silent mode, you can find OTN article at below mentioned link

http://www.oracle.com/technology/pub/articles/revitt-oem.html

There is also metalink Note 763347.1 – How to Install Enterprise Manager Grid Control 10.2.0.5.0 Using an Existing Database with the Software Only Method

One of the issue while using silent install mode is that due to Bug 7137054 (Refer Note 602750.1 ConfigureGC.pl Reports – Invalid Username/Password ), you will be required to change sys password and run the ConfigureGC.pl script multiple times. Also you would be required to enter the password’s in the response file, which you might not be comfortable with. (Yeah in some environments, people can be very particular 🙂 )

While using GUI mode, I did not face any issues arising out of bug 7137054. Below steps are tried on a 10.2.0.4 database (11g database can also be used) and Platform is AIX 5L. Installation steps would remain same for other platforms too (In windows you will have to check the steps where in you are supposed to run scripts. GUI part should remain same)

Steps To Perform 10.2.0.5 EM Grid Installation using Software-only Install Method – GUI

1. Ensure that  database parameters are set to following values. This is important as OUI checks for these values and if it is not set , then installation fails

job_queue_processes 10
db_block_size 8192
timed_statistics TRUE
open_cursors 300
session_cached_cursors 200
aq_tm_processes 1
compatible <currently installed Oracle Database release> (default)
undo_management AUTO
undo_retention 10800
undo_tablespace <any acceptable name>
processes 150
log_buffer 1048576
statistics_level TYPICAL (Note that this value is specific only to Enterprise Manager 10g Repository Database release and later.)
TEMP space (Tablespace)Foot 1 50 MB (extending to 100 MB)
_b_tree_bitmap_plans false (hidden parameter)

2. Ensure that dbconsole repository is not present in the database. If it is, then remove it. You need to stop the dbconsole before starting the process.

 $ORACLE_HOME/bin/emctl stop dbconsole

Remove the following directories from your filesystem:

rm -rf $ORACLE_HOME/(hostname_sid)
rm -rf $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_(hostname)_(sid)

Removing DB Repository

ALTER system ENABLE RESTRICTED SESSION ;
EXEC sysman.emd_maintenance.remove_em_dbms_jobs;
EXEC sysman.setEMUserContext('',5);
REVOKE dba FROM sysman;
DECLARE CURSOR c1 IS SELECT owner, synonym_name name FROM dba_synonyms WHERE table_owner = 'SYSMAN';
 BEGIN
FOR r1 IN c1
 LOOP
IF r1.owner = 'PUBLIC'
THEN
 EXECUTE IMMEDIATE 'DROP PUBLIC SYNONYM '||r1.name; ELSE EXECUTE IMMEDIATE 'DROP SYNONYM '||r1.owner||'.'||r1.name;
END IF;
 END LOOP;
 END;
/
DROP USER mgmt_view CASCADE;
 DROP ROLE mgmt_user;
DROP USER sysman CASCADE;
 ALTER SYSTEM DISABLE RESTRICTED SESSION;

3.Log in to the database as SYSDBA and run the following command to install DBMS_SHARED_POOL package:

@?/rdbms/admin/dbmspool.sql

commit;

4. Install the 10.2.0.1 database using -noconfig option i.e to skip the configuration part. This will be carried out in the end by running ConfigureGC.pl script

./runInstaller -noconfig

Choose Installation with Existing Database

Specify the ORACLE_BASE Location under which you would like to install the EM Grid

You will be now prompted to enter the database details for the Repository database. Ensure that listener is UP.

a)In the Configure Email Notification section, specify an appropriate e-mail address, and the corresponding SMTP server name. You will receive important information on the condition of the monitored targets, including critical alerts at this e-mail address.

b)Specify the My Oracle Support (formerly Metalink) credentials.

c)Specify the Proxy Information if Grid Control is using a proxy server for external access


Refer to following documentation link for more information

http://download.oracle.com/docs/cd/B16240_01/doc/install.102/e10953/installing_em.htm#CHDJJBCJ

a)Specify the password for Securing agent
b)Specify password for SYSMAN user. Please note that default ias_admin password is the same as the password assigned to the SYSMAN account. This is required to access the Oracle application server console



OUI will install OMS under $BASE_DIR/oms10g

OUI will install Management Agent under $BASE_DIR/agent10g

Once the installation finishes , you will be prompted to run allroot.sh script (as shown below). Before executing it, stop all the OPMN processes by running the following command from the Oracle home directory of the OMS (for example, oms10g):

Open a different shell and execute following commands using Oracle Software owner

Processes in Instance: EnterpriseManager0.db11g
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
DSA                | DSA                |     N/A | Down
HTTP_Server        | HTTP_Server        |     N/A | Down
LogLoader          | logloaderd         |     N/A | Down
dcm-daemon         | dcm-daemon         |     N/A | Down
OC4J               | home               |     N/A | Down
WebCache           | WebCache           |     N/A | Down
WebCache           | WebCacheAdmin      |     N/A | Down

$./opmnctl stopall
opmnctl: stopping opmn and all managed processes...

$./opmnctl status
Unable to connect to opmn.
Opmn may not be up.

Now run the script allroot.sh as Root.


You will get a message that configuration assistants were skipped. You can ignore it as we used -noconfig method



Installation for 10.2.0.1 Base release is complete now. We now need to proceed towards 10.2.0.5 Patchset installation


5. Run following command from 10.2.0.5 patchset directory.

<span style="font-size: small;"> </span>cd $patch_dir/3731593/Disk1
./runInstaller -noconfig b_softwareonly=true

Choose OMS home to patch and proceed


You will be prompted for Sys password


Enter ias_admin Password. This will be same as SYSMAN password


Application server console will be now shutdown. You will get a message confirming same. Click Ok.

After Installation completes , you will be prompted to run root.sh (as shown below) as root from OMS_HOME


6. Next apply 10.2.0.5 patchset to AGENT_HOME by issuing following command.

<span style="font-size: small;">.</span>/runInstaller -noconfig

Select AGENT_HOME for patching i.e agent10g


After Installation completes , you will be prompted to Run root.sh (as shown below) from Oracle Home for Management Agent ($BASE_DIR/agent10g)


7.Now the installation part is over. Next we need to run ConfigureGC.pl script which will actually configure 10.2.0.5 Grid Control.

Before making changes to the environment variable PERL5LIB, take a backup of the variable:

 setenv PERL5LIB_BACKUP $PERL5LIB

Set PERL5LIB to $OMS_ORACLE_HOME/perl/lib/5.6.1

export PERL5LIB=/db40/temp2/oms10g/perl/lib/5.6.1

Configure Grid Control by running the ConfigureGC.pl script from the Oracle home directory of the OMS

/db40/temp2/oms10g/perl/bin/perl /db40/temp2/oms10g/sysman/install/ConfigureGC.pl /db40/temp2

This script will take long time (took 2 hrs on my setup). You can monitor $OMS_ORACLE_HOME/cfgtoollogs/cfgfw/ CfmLogger*log file to check for any errors

The plug-in Starting Oracle Management Server has successfully been performed
------------------------------------------------------
------------------------------------------------------
The plug-in EMCLI Configuration is running

Operation EMCLI Setup is in progress.

The plug-in EMCLI Configuration has successfully been performed
------------------------------------------------------
The action patchsetConfiguration has successfully completed
###################################################

Once you get message that the configuration is complete, then check the status for OMS and AGENT

$OMS_ORACLE_HOME/bin/emctl status oms
$AGENT_ORACLE_HOME/bin/emctl status agent

You can now proceed with Agent installation on the targets to be monitored

Link: Best practices for AIX RAC Database on OTN

A new whitepaper has been published on OTN which primarily discusses best practices for AIX 5.2,5.3 and 6.1 for RAC database for avoiding node eviction due to Oprocd.  It also mentions recommended AIX VMO parameters and recommended patches for Oracle RAC.

You can find the article at below location

http://www.oracle.com/technology/products/database/clusterware/pdf/rac_aix_system_stability.pdf

Creating ASM devices on AIX

I thought of sharing few tips on creating ASM devices on AIX which I will be helpful to Oracle DBA’s. Suppose SysAdmin gives you list of Serial numbers for LUN instead of device Name

pcmpath query device
DEV#:  33  DEVICE NAME: hdisk33  TYPE: 2107900  ALGORITHM:  Load Balance
SERIAL: 75DM011<span style="color: #ff0000;"><strong>1101</strong></span>
===========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0           fscsi0/path0          CLOSE   NORMAL          9          0
    1           fscsi1/path1          CLOSE   NORMAL          8          0

In case there are lot many disks, then it could be a tiring task of running above command and finding each device. You can use below code which will list name of  devices and size (In MB) of disk.

for i in 1000 1100    1018    1118    1030    1130    104C    114C    1068    1168    1080    1180
do
j=`pcmpath query device|grep -p $i"$"|grep DEVICE|awk -F ":" '{print }'|awk '{print }`
k=`bootinfo -s $j`
echo $i $j $k
done

This would return following output

1000 hdisk4 65536
1100 hdisk10 65536
1018 hdisk5 65536
1118 hdisk11 65536
1030 hdisk6 65536
1130 hdisk12 65536
104C hdisk7 65536
114C hdisk13 65536
1068 hdisk8 65536
1168 hdisk14 65536
1080 hdisk9 65536
1180 hdisk15 65536

Now if you need to create new device name, you need to use mknod command and pass on major and minor numbers. Following code can be used to perform same

#export m=0
 # for i in hdisk4  hdisk10 hdisk5  hdisk11 hdisk6  hdisk12 hdisk7  hdisk13 hdisk8  hdisk14 hdisk9  hdisk15
 do
 j=`ls -la /dev/$i |awk '{print }'|awk -F "," '{print }'`
  k=`ls -la /dev/$i |awk '{print }'`
 m=`expr $m + 1` ;echo "mknod /dev/asm_disk"$m "c "$j $k
 done

 mknod /dev/asm_disk1 c 21 4
 mknod /dev/asm_disk2 c 21 12
 mknod /dev/asm_disk3 c 21 13
 mknod /dev/asm_disk4 c 21 15
 mknod /dev/asm_disk5 c 21 5
 mknod /dev/asm_disk6 c 21 6
 mknod /dev/asm_disk7 c 21 8
 mknod /dev/asm_disk8 c 21 7
 mknod /dev/asm_disk9 c 21 14
 mknod /dev/asm_disk10 c 21 10
 mknod /dev/asm_disk11 c 21 9
 mknod /dev/asm_disk12 c 21 11

Now you can change the ownership to oracle:dba and permission to 660. I have 12 disks , so using list of 12 variables. In case you have more disks , then you can add more variables

# for i in 1 2 3 4 5 6 7 8 9 10 11 12
do
chown oracle:dba /dev/asm_disk$i
chmod 660 /dev/asm_disk$i
done

crw-rw----    1 oracle   dba          21, 11 Jan 28 17:10 /dev/asm_disk12
crw-rw----    1 oracle   dba          21,  9 Jan 28 17:10 /dev/asm_disk11
crw-rw----    1 oracle   dba          21, 10 Jan 28 17:10 /dev/asm_disk10
crw-rw----    1 oracle   dba          21, 14 Jan 28 17:04 /dev/asm_disk9
crw-rw----    1 oracle   dba          21,  7 Jan 28 17:04 /dev/asm_disk8
crw-rw----    1 oracle   dba          21,  8 Jan 28 17:04 /dev/asm_disk7
crw-rw----    1 oracle   dba          21,  6 Jan 28 17:04 /dev/asm_disk6
crw-rw----    1 oracle   dba          21,  5 Jan 28 17:04 /dev/asm_disk5
crw-rw----    1 oracle   dba          21, 15 Jan 28 17:04 /dev/asm_disk4
crw-rw----    1 oracle   dba          21, 13 Jan 28 17:04 /dev/asm_disk3
crw-rw----    1 oracle   dba          21, 12 Jan 28 17:04 /dev/asm_disk2
crw-rw----    1 oracle   dba          21,  4 Jan 28 17:04 /dev/asm_disk1

In case you need to use same logic for creating OCR and Voting disks on RAC system, replace /dev/asm with /dev/ocr or /dev/voting . I hope this would save some time and also prevent errors 🙂

In case you have disks in ordered number,say 53 to 62 then you can also use for loop as below.

#bash
bash-3.00#
#export m=0
#for ((i=53;i<=62;i++))
do
 j=`ls -la /dev/hdisk$i |awk '{print }'|awk -F "," '{print }'`
  k=`ls -la /dev/hdisk$i |awk '{print }'`
   m=`expr $m + 1` ;echo "mknod /dev/asm_disk"$m "c "$j $k
 done
#for ((i=1;i<=10;i++))
do
chown oracle:dba /dev/asm_disk$i
chmod 660 /dev/asm_disk$i
done

I would suggest anyone using the scripts to first check in a test environment.

CRS Fails to Start – 10.2.0.1 RAC Install on AIX

I was installing 10.2.0.1 on IBM AIX 5L and while running root.sh from first node (as part of Clusterware installation) got following messages

Now formatting voting device: /dev/voting_disk01
Now formatting voting device: /dev/voting_disk02
Now formatting voting device: /dev/voting_disk03
Format of 3 voting devices complete.
Startup will be queued to init within 30 seconds

I waited for quite some time and found that it was stuck. To check what was status of CSS, I did a grep for CSS and found that it was running /etc/init.cssd startcheck css script. This indicated that Oracle was stuck trying to start CSS. Following errors were recorded in /tmp/crsct.7459

Failure in CSS initialization opening OCR.

Metalink notes suggested checking OCR Disk permission , though in my case they had correct permissions i.e ownership as oracle:dba and permission set to 660. To diagnose further, I checked $ORA_CRS_HOME/log to check for errors. All the logfiles related to CRS,CSS and EVMD are stored in $ORA_CRS_HOME/log/<hostname>.

/oracle/crs_base/app/product/crs10gR2/log>ls -ltr
total 0
drwxrwx---    2 oracle   dba             256 Jan 28 18:46 crs
drwx------    3 root     system          256 Jan 28 18:53 chd0196
drwxr-xr-t    8 root     dba             256 Jan 28 18:53 rac01

Hostname for the server was rac01 and not chd0196. This was a new server and also directories could not be present earlier as it was a fresh installation.  Oracle was picking two hostname which was quite strange. I checked for HACMP filesets  and found that they were present

# lslpp -l |grep -i hacmp
  rsct.basic.hacmp           2.4.9.0  COMMITTED  RSCT Basic Function (HACMP/ES
  rsct.compat.basic.hacmp    2.4.9.0  COMMITTED  RSCT Event Management Basic
                                                 Function (HACMP/ES Support)
  rsct.compat.clients.hacmp  2.4.9.0  COMMITTED  RSCT Event Management Client
                                                 Function (HACMP/ES Support)

10g RAC does not require Vendor clusterware as Oracle provides it own clusterware called “Oracle Clusterware”.We got these packages un-installed and got both server rebooted. After cleaning up RAC installation, we restarted installation . You can use  Metalink Note 239998.1 – 10g RAC: How to Clean Up After a Failed CRS Install for cleanup.  On re-running root.sh installation, installation went fine.

10.2.0.4 on AIX5L (64-Bit) is Out

As usual Laurent has been first to locate that 10.2.0.4 on AIX5L (64-Bit) is Out.

Patchset size is whopping 1.9G which I believe is the maximum size for Oracle Patchset till date. Looks like 10.2.0.3 on AIX had lot of bugs or it could be that DATABASE REPLAY feature needed more lines of code in AIX 🙂

I am not sure why Oracle has named this feature as DATABASE REPLAY on 10.2.0.4. It should have been named as DATABASE CAPTURE.

Anyways I was trying to be the first person to post release of 10.2.0.4 Patchset , but I guess Laurent has written some Package which sends a SMS that patchset is out 🙂 I need to start learning PL/SQL.