Unix

Scheduling Job through Cron??

In case you came across this link searching for Cron syntax, then you can refer to following link

http://www.adminschoice.com/docs/crontab.htm

Other’s can continue reading the article …. 😉

This was a problem which I faced yesterday. We had a mview refresh which was lagging on one of the server. On checking master site, I found that there were many pending requests coming from the second materialized view site . Checking second site revealed that there were multiple sessions running in database which were waiting on ‘enq :JI Contention’ wait event.

These mviews were being refreshed by job scheduled through a cron. Doing a grep for the refresh script gave following result

 $ps -ef|grep scott|grep -v grep
    oracle 11725 11714  0 13:00:00 ?         0:00 /usr/bin/sh /home/oracle/cron/Refresh_scott.sh
  oracle 19981 19970  0  Feb  7  ?         0:00 /usr/bin/sh /home/oracle/cron/Refresh_scott.sh
  oracle 24794 24781  0 06:30:00 ?         0:00 /usr/bin/sh /home/oracle/cron/Refresh_scott.sh
    oracle 10538 10527  0  Feb  6  ?         0:00 /usr/bin/sh /home/oracle/cron/Refresh_scott.sh
    oracle 13972 13935  0  Feb  6  ?         0:00 /usr/bin/sh /home/oracle/cron/Refresh_scott.sh
  oracle  2601  2592  0 19:00:00 ?         0:00 /usr/bin/sh /home/oracle/cron/Refresh_scott.sh
  oracle 17274 17246  0  Feb  6  ?         0:00 /usr/bin/sh /home/oracle/cron/Refresh_scott.sh
  oracle  8308  8294  0 13:30:00 ?         0:00 /usr/bin/sh /home/oracle/cron/Refresh_scott.sh

This was definitely wrong. I killed the database sessions and also killed the OS processes

$ ps -ef|grep Refresh_scott.sh|grep -v grep|awk ‘{print $2}’|xargs kill -9

This issue had occurred as one more site was added for mview refresh and due to locking issues (while deleting records from Mlog$), job could not complete before next refresh schedule. While scheduling scripts through cron, care should be taken to put a check if script is already running or not. This is required as Cron will spawn the new job (as per job schedule) irrespective of fact that whether earlier execution has completed or not. I used following logic to implement the same check

cnt=`ps -ef|grep Refresh_scott.sh|grep -v grep |wc -l`
if [ $cnt -eq 1 ]
then
echo "Running Fast Refresh now " > $DIR/Check_refresh_scott.log
sqlplus /nolog <<EOF > $DIR/Refresh_scott.log
conn scott/tiger
exec dbms_mview.refresh ...
...
exit
EOF
else
echo "Fast refresh already running " > $DIR/Check_refresh_scott.log
date >> $DIR/Check_refresh_scott.log
exit
fi

There could be other logics too which can be used to implement the same. e.g Other could be

filename=$DIR/Check_refresh_scott.log
if [ -e $filename  ]
then
echo "Fast Refresh already running " > refresh_runnning.log
exit
else
sqlplus /nolog <<EOF > $DIR/Refresh_scott.log
conn scott/tiger
exec dbms_mview.refresh ...
...
exit
EOF
rm $filename
fi

Other way could be to use DBMS_JOB and DBMS_SCHEDULER to schedule the database job.

Creating ASM devices on AIX

I thought of sharing few tips on creating ASM devices on AIX which I will be helpful to Oracle DBA’s. Suppose SysAdmin gives you list of Serial numbers for LUN instead of device Name

pcmpath query device
DEV#:  33  DEVICE NAME: hdisk33  TYPE: 2107900  ALGORITHM:  Load Balance
SERIAL: 75DM011<span style="color: #ff0000;"><strong>1101</strong></span>
===========================================================================
Path#      Adapter/Path Name          State     Mode     Select     Errors
    0           fscsi0/path0          CLOSE   NORMAL          9          0
    1           fscsi1/path1          CLOSE   NORMAL          8          0

In case there are lot many disks, then it could be a tiring task of running above command and finding each device. You can use below code which will list name of  devices and size (In MB) of disk.

for i in 1000 1100    1018    1118    1030    1130    104C    114C    1068    1168    1080    1180
do
j=`pcmpath query device|grep -p $i"$"|grep DEVICE|awk -F ":" '{print }'|awk '{print }`
k=`bootinfo -s $j`
echo $i $j $k
done

This would return following output

1000 hdisk4 65536
1100 hdisk10 65536
1018 hdisk5 65536
1118 hdisk11 65536
1030 hdisk6 65536
1130 hdisk12 65536
104C hdisk7 65536
114C hdisk13 65536
1068 hdisk8 65536
1168 hdisk14 65536
1080 hdisk9 65536
1180 hdisk15 65536

Now if you need to create new device name, you need to use mknod command and pass on major and minor numbers. Following code can be used to perform same

#export m=0
 # for i in hdisk4  hdisk10 hdisk5  hdisk11 hdisk6  hdisk12 hdisk7  hdisk13 hdisk8  hdisk14 hdisk9  hdisk15
 do
 j=`ls -la /dev/$i |awk '{print }'|awk -F "," '{print }'`
  k=`ls -la /dev/$i |awk '{print }'`
 m=`expr $m + 1` ;echo "mknod /dev/asm_disk"$m "c "$j $k
 done

 mknod /dev/asm_disk1 c 21 4
 mknod /dev/asm_disk2 c 21 12
 mknod /dev/asm_disk3 c 21 13
 mknod /dev/asm_disk4 c 21 15
 mknod /dev/asm_disk5 c 21 5
 mknod /dev/asm_disk6 c 21 6
 mknod /dev/asm_disk7 c 21 8
 mknod /dev/asm_disk8 c 21 7
 mknod /dev/asm_disk9 c 21 14
 mknod /dev/asm_disk10 c 21 10
 mknod /dev/asm_disk11 c 21 9
 mknod /dev/asm_disk12 c 21 11

Now you can change the ownership to oracle:dba and permission to 660. I have 12 disks , so using list of 12 variables. In case you have more disks , then you can add more variables

# for i in 1 2 3 4 5 6 7 8 9 10 11 12
do
chown oracle:dba /dev/asm_disk$i
chmod 660 /dev/asm_disk$i
done

crw-rw----    1 oracle   dba          21, 11 Jan 28 17:10 /dev/asm_disk12
crw-rw----    1 oracle   dba          21,  9 Jan 28 17:10 /dev/asm_disk11
crw-rw----    1 oracle   dba          21, 10 Jan 28 17:10 /dev/asm_disk10
crw-rw----    1 oracle   dba          21, 14 Jan 28 17:04 /dev/asm_disk9
crw-rw----    1 oracle   dba          21,  7 Jan 28 17:04 /dev/asm_disk8
crw-rw----    1 oracle   dba          21,  8 Jan 28 17:04 /dev/asm_disk7
crw-rw----    1 oracle   dba          21,  6 Jan 28 17:04 /dev/asm_disk6
crw-rw----    1 oracle   dba          21,  5 Jan 28 17:04 /dev/asm_disk5
crw-rw----    1 oracle   dba          21, 15 Jan 28 17:04 /dev/asm_disk4
crw-rw----    1 oracle   dba          21, 13 Jan 28 17:04 /dev/asm_disk3
crw-rw----    1 oracle   dba          21, 12 Jan 28 17:04 /dev/asm_disk2
crw-rw----    1 oracle   dba          21,  4 Jan 28 17:04 /dev/asm_disk1

In case you need to use same logic for creating OCR and Voting disks on RAC system, replace /dev/asm with /dev/ocr or /dev/voting . I hope this would save some time and also prevent errors 🙂

In case you have disks in ordered number,say 53 to 62 then you can also use for loop as below.

#bash
bash-3.00#
#export m=0
#for ((i=53;i<=62;i++))
do
 j=`ls -la /dev/hdisk$i |awk '{print }'|awk -F "," '{print }'`
  k=`ls -la /dev/hdisk$i |awk '{print }'`
   m=`expr $m + 1` ;echo "mknod /dev/asm_disk"$m "c "$j $k
 done
#for ((i=1;i<=10;i++))
do
chown oracle:dba /dev/asm_disk$i
chmod 660 /dev/asm_disk$i
done

I would suggest anyone using the scripts to first check in a test environment.

CRS Fails to Start – 10.2.0.1 RAC Install on AIX

I was installing 10.2.0.1 on IBM AIX 5L and while running root.sh from first node (as part of Clusterware installation) got following messages

Now formatting voting device: /dev/voting_disk01
Now formatting voting device: /dev/voting_disk02
Now formatting voting device: /dev/voting_disk03
Format of 3 voting devices complete.
Startup will be queued to init within 30 seconds

I waited for quite some time and found that it was stuck. To check what was status of CSS, I did a grep for CSS and found that it was running /etc/init.cssd startcheck css script. This indicated that Oracle was stuck trying to start CSS. Following errors were recorded in /tmp/crsct.7459

Failure in CSS initialization opening OCR.

Metalink notes suggested checking OCR Disk permission , though in my case they had correct permissions i.e ownership as oracle:dba and permission set to 660. To diagnose further, I checked $ORA_CRS_HOME/log to check for errors. All the logfiles related to CRS,CSS and EVMD are stored in $ORA_CRS_HOME/log/<hostname>.

/oracle/crs_base/app/product/crs10gR2/log>ls -ltr
total 0
drwxrwx---    2 oracle   dba             256 Jan 28 18:46 crs
drwx------    3 root     system          256 Jan 28 18:53 chd0196
drwxr-xr-t    8 root     dba             256 Jan 28 18:53 rac01

Hostname for the server was rac01 and not chd0196. This was a new server and also directories could not be present earlier as it was a fresh installation.  Oracle was picking two hostname which was quite strange. I checked for HACMP filesets  and found that they were present

# lslpp -l |grep -i hacmp
  rsct.basic.hacmp           2.4.9.0  COMMITTED  RSCT Basic Function (HACMP/ES
  rsct.compat.basic.hacmp    2.4.9.0  COMMITTED  RSCT Event Management Basic
                                                 Function (HACMP/ES Support)
  rsct.compat.clients.hacmp  2.4.9.0  COMMITTED  RSCT Event Management Client
                                                 Function (HACMP/ES Support)

10g RAC does not require Vendor clusterware as Oracle provides it own clusterware called “Oracle Clusterware”.We got these packages un-installed and got both server rebooted. After cleaning up RAC installation, we restarted installation . You can use  Metalink Note 239998.1 – 10g RAC: How to Clean Up After a Failed CRS Install for cleanup.  On re-running root.sh installation, installation went fine.

Input is too long (> 2499 characters) – line ignored

There are times when you observe above mentioned error while creating view and mview from mview definition taken from Toad or indexfile option. vi editor comes in handy to resolve this issue. Copy the code in Text file and while in escape mode enter following command

:%s/,/,^M/g

Please note that to insert ^M you need to press Ctrl+V+M keys together.We are basically using the command to enter a line break after each comma. 

Effect of OS Terminal Setting “STTY” on Oracle Database

Few days back, while trying to create a synonym over dblink I experienced following error:

SQL> create synonym synonymn_name for dual@DBLINK9i;
SP2-0042: unknown command "DBLINK9i" - rest of line ignored

Strange error !!!

Also while trying to perform a simple select statement over another dblink:

SQL > select * from dual@dblink10g;
SP2-0042: unknown command "DBLINK10g" - rest of line ignored.

It also failed with a similar error, though the select over this dblink was working fine earlier.The tnsnames.ora entry was correct and double checked.So what happened to this dblink now?

From other database I was able to perform a simple select over same dblink.

While I was researching this, I came to know that sql queries also started failing :

SQL> UPDATE HRRECORD_TEST SET EMAIL_ADDRESS = '[email protected]' WHERE NAME = 'XYZ';
SP2-0734: unknown command beginning "123.com'..." - rest of line ignored.
SQL>

One Notable thing was that all the queries which failed contained “@” symbol and I also got this error  while create a synonym over dblink (having “@” symbol).

These two issues were related, Then I tried to create a synonym locally and it went fine. This led to conclusion  that sql prompt was not recognising “@” symbol.

So it was clear that the problem was with terminal settings at OS level for oracle user.

Searcing google for terminal setting at OS, I found a command which is used to Sets options for your terminal i.e “stty”. By using stty we can change the terminal settings.

Now checking the current terminal settings using:

# stty -a

speed 9600 baud; line = 0;
rows = 36; columns = 80
min = 4; time = 0;
intr = DEL; quit = ^\; erase = DEL; kill = @
eof = ^D; eol = ^@; eol2 <undef>; swtch <undef>
stop = ^S; start = ^Q; susp <undef>; dsusp <undef>
werase <undef>; lnext <undef>
parenb -parodd cs7 -cstopb hupcl -cread -clocal -loblk -crts

From this output I found that there were two settings causing problem:

1) kill = @

2) eol = ^@

Then, to overcome this I added the following two lines in the oracle user .profile

stty kill ^U

stty eol ^E

Above change will force oracle user to use ^U to kill any session and ^E to end a line instead of using @ for these purpose.

After making these changes in oracle user’s .profile everything went fine. In the end it turned out OS issue whose settings was modified by sysadmin. 😉

Easy and Free X-Windowing Setup with Cygwin

Today I had to carry out Oracle client Installation on one of the server. Normally I use vncviewer to connect to one of my Linux machine (having vncserver setup) and then get the display on to that machine.

But my colleague suggested me about one more approach, which I thought of sharing with all of you. This is using a X –Windowing software which is Free 🙂 Surprised!!

This can be done using Cygwin which is a freeware so you can trash your Reflection X or Hummingbird Installations. You can check various X-Windows software available here

Lets come back to Cygwin. You can download the software from http://x.cygwin.com/

To perform the Cygwin setup, you need to follow below steps

1) Start the XWindows server on your PC. This could be done by running startxwin-multiwindow file present on Desktop (Created during installation) or running /usr/X11R6/bin/startxwin.sh from cygwin

$ sh /usr/X11R6/bin/startxwin.sh

You can also use startx. For configuration check this link

http://x.cygwin.com/docs/ug/using.html

3) Connect to the Unix/Linux machine and set the display to your machine. Use ipconfig command to get the IP address for your machine. E.g

C:\Documents and Settings\amit &gt;ipconfig
Windows IP Configuration
Ethernet adapter Local Area Connection:
        Connection-specific DNS Suffix  . : xyz.com
        IP Address. . . . . . . . . . . . : <strong>192.168.4.47</strong>
        Subnet Mask . . . . . . . . . . . : X.X.X.X
        Default Gateway . . . . . . . . . : X.X.X.X

Now set the DISPLAY variable as

$export DISPLAY=192.168.4.47:0.0

4) Test it by typing any application like xcalc,xclock,etc.

If X11 Forwarding is enabled on the remote machine, you can also directly login to server using ssh

$ssh oracle@ip –X

Now you can carry out oracle Installation using runInstaller 🙂