oracle

CleanUp Temporary Segments Occupying Permanent Tablespace

There are situations when we see “temporary segments” in permanent tablespaces hanging around and not getting cleaned up.
These temporary segments in permanent tablespace can be created by DDL operations like CTAS and “alter index..rebuild” because
the new object is created as a temporary segment in the target tablespace and when the DDL action finishes it will be changed to permanent type.
These temporary segments take actual disk space when SMON fails to perform its assigned job to cleanup stray temporary segments.
Following query finds out these segments:

 SQL > select tablespace_name, owner, sum(bytes/1024/1024) from dba_segments
where segment_type = 'TEMPORARY' group by tablespace_name, owner;

TABLESPACE_NAME     OWNER          SEGMENT_NAME     SUM(BYTES/1024/1024)
------------------- ------------   ------------     --------------------
xxxx_DATA           SYS              123.8365          137706
BDEPST_INDEX        SYS              345.8756            8910
KMRPT_DATA          SYS                345.87       25284.875
BILL_INDEX          SYS                                   .25
DSS_DATA            SYS                                   798
MRKT_INDEX          SYS                                   208
SPCT_DATA           SYS                              69642.25
SPCT_INDEX          SYS                              956.4375

Here we can see that tablespace KMRPT_DATA, SPCT_INDEX and SPCT_DATA have large temporary segments.

To know if any DDL is active which can create temporary segments we can use the following:

SQL> conn / as sysdba
SQL> select owner FROM dba_segments WHERE segment_name='345.87';
SQL> select pid from v$process where username='owner from above query';
SQL> alter session set tracefile_identifier='TEMPORARY_SEGMENTS';
SQL> oradebug setorapid <pid obtained>
SQL> oradebug dump errorstack 3
SQL > oradebug tracefile_name

It will give you the tracefile name, open that file and check for the “current sql”
If it is a DDL like CTAS or index rebuild, then wait for the operation to complete. If there is no pid
returned then these segments are “stray segements” and needs to cleaned up manually.

There are two ways to force the drop of temporary segments:

1. Using event DROP_SEGMENTS
2. Corrupting the segments and dropping these corrupted segments.

1. Using DROP_segments:

Find out the tablespace number (ts#) which contains temporary segments:
SQL> select ts# from sys.ts$ where name = 'tablespace name';

Suppose it comes out to be 10, use the following command to cleanup temporary segments:

SQL> alter session set events 'immediate trace name DROP_SEGMENTS level 11';

level is ts#+1 i.e 10+1=11 in this case.

2. Corrupting temporary segments for drop:
For this following procedures are used:
– DBMS_SPACE_ADMIN.TABLESPACE_VERIFY
– DBMS_SPACE_ADMIN.SEGMENT_CORRUPT
– DBMS_SPACE_ADMIN.SEGMENT_DROP_CORRUPT

— Verify the tablespace that contains temporary segments (In this case it is KMRPT_DATA)

SQL>DBMS_SPACE_ADMIN.TABLESPACE_VERIFY('KMRPT_DATA');

— Corrupt the temporary segments in tablespace KMRPT_DATA

SQL>DBMS_SPACE_ADMIN.SEGMENT_CORRUPT(' || chr(39) || tablespace_name || chr(39) || ',' || HEADER_FILE || ',' || HEADER_BLOCK || ');'  from dba_segments where SEGMENT_TYPE like 'TEMP%' and tablespace_name = 'KMRPT_DATA';

— Drop the corrupted temporary segments

SQL> select 'exec DBMS_SPACE_ADMIN.SEGMENT_DROP_CORRUPT (' || chr(39) || tablespace_name || chr(39) || ',' || HEADER_FILE || ',' || HEADER_BLOCK || ');' from dba_segments where SEGMENT_TYPE like 'TEMP%' and tablespace_name = 'KMRPT_DATA';

— Verify the tablespace again to update the new dictionary information:

SQL>DBMS_SPACE_ADMIN.TABLESPACE_VERIFY('KMRPT_DATA');

This will remove temporary segments from permanent tablespace.

Cheers!!!

Saurabh Sood

Link: Best practices for AIX RAC Database on OTN

A new whitepaper has been published on OTN which primarily discusses best practices for AIX 5.2,5.3 and 6.1 for RAC database for avoiding node eviction due to Oprocd.  It also mentions recommended AIX VMO parameters and recommended patches for Oracle RAC.

You can find the article at below location

http://www.oracle.com/technology/products/database/clusterware/pdf/rac_aix_system_stability.pdf

Cloning database..Just do some Post checks !!

Well I will be covering some points which I believe should be part of post checklist for any cloned database environment. This might not be complete list as it contains point which I have encountered or heard of.

1) Change database Name and Database Id

You should try to change the database name/instance name for UAT/ Cloned environment. In case if there is specific requirement to have the same instance_name/db_name, then atleast you should try changing the database id. If you are using RMAN duplicate command, dbid will be changed automatically. But in case you do not use duplicate command, then dbid will remain same.

DBid change becomes very important if you are using rman catalog database. In case you connect to rman catalog database from new cloned DB (without changing DBID),it would resync the resetlog information in the rman catalog database. Next time you try to take RMAN database backup on production database, you will get following errors

RMAN-06004: ORACLE error from recovery catalog database:
RMAN-20011: target database incarnation is not current in recovery catalog
 

 Following metalink notes discuss how to handle these issues
 
Note 1070453.6 : RMAN: Point-in-Time Recovery of a Backup From Before Last Resetlogs
Note 237232.1 : How to Recover Through a Resetlogs Command Using RMAN


As you see, you will be better off changing Dbid (Database Id) at first place. DBNEWID can be used to change the DB_NAME or DBId or Both

http://download.oracle.com/docs/cd/B19306_01/server.102/b14215/dbnewid.htm

2) Check for database link

Check for database link present in the cloned environment. Ensure that these are select only dblinks and will not perform any DML in production databases. If you find any ,then you can either drop these or recreate them to point to any UAT or simply remove/hash out tnsnames entry corresponding to these hosts. Also check for any hard coded IP address in host column in DBA_DB_LINKS.

3)Remove or hash out any entries in tnsnames.ora pointing to production database and recovery catalog database

This again is to avoid any issues with dblink or RMAN catalog issues. Also note that when you hash the tnsnames.ora you need to place # in front of each line.

Incorrect Correct
#TESTDB10G =
(DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = test10g)(PORT = 1521))
      )
      (CONNECT_DATA =
      (SERVICE_NAME = testdb10g)
    )
 )
#TESTDB10G =
#(DESCRIPTION =
#   (ADDRESS_LIST =
#      (ADDRESS = (PROTOCOL = TCP)(HOST = test10g)(PORT = 1521))
 #     )
#    (CONNECT_DATA =
#     (SERVICE_NAME = testdb10g)
#   )
#)

 

 

 

4) Check cronjobs for oracle database user
If you have copied cron entries for server from production ( as part of database migration activity) , then crosscheck which all jobs need to be enabled for this cloned environment. e.g You can easily disable any cron for RMAN database, archivelog backup.

5)Modify listener.ora file

Modify listener.ora file to include new host entry and port number. This is also quite important as in case you copied it from production server, you could turn off production listener by mistake. Check following metalink note for details

Note 460666.1 – How To Remotely Administer a Listener

6) Check for local_listener and remote_listener parameter

Check for local_listener and remote_listener parameter and modify accordingly. Note that not changing remote_listener parameter can also lead to issues where in your UAT/Test database can get registered with Listener running on Production server.

7)Modify /etc/hosts entries

Modify /etc/hosts entries to remove entries for production database. Also try to use /etc/hosts for resolving host instead of DNS.

8) Add tempfiles to Temporary tablespace

After you clone the environment, tempfiles need to be added to the database.

9) Check for Archivelog mode

Generally archivelog mode is disabled for UAT/Cloned databases. In case your production database is in archivelog mode, ensure that you disable the archiving. 

10) Verify Initialization Parameters

Verify all initialization parameter’s like *_dump_dest locations,utl_file_dir ,sga_max_size , etc.

Silent Install :OUI-10133:Invalid staging area Error

While using silent install for database software installation , you could encounter following errors

SEVERE:OUI-10133:Invalid staging area. There are no top level components for IBM SP AIX available for installation in this staging area.

This error is reported when OUI is unable to find products.xml file. Check response file for FROM_LOCATION parameter and see if it is correct and Oracle software owner has read permission on this file. In my case it was set to /oracle/dump/Disk1/database/stage/products.xml but the actual file was present at /oracle/ora10g/Disk1/database/stage/products.xml

ls -l "/oracle/ora10g/Disk1/database/stage/products.xml"
-rwxr-xr-x    1 ora10g   oinstall     804962 Aug 20 2005  /oracle/ora10g/Disk1/database/stage/products.xml

Modify the FROM_LOCATION to correct value and re-run the installation. It should go fine.

This parameter can have incorrect settings when you have copied the response file from $media/response/*rsp to a local directory due to which relative file naming convention makes file unavailable. By default value for FROM_LOCATION is set to “../stage/products.xml”. Edit the response file and enter the absolute path giving correct location for the file.

It can also be the case that you have created a custom response file using runInstaller GUI mode and the software media directory has now changed on the new server on which this response file is being used. e.g In my case response file was created when the Oracle software was staged in /oracle/dump/Disk1 directory but on the new server it has been moved to /oracle/ora10g/Disk1 . Due to this , OUI was not able to locate products.xml file and we were required to change value for FROM_LOCATION variable.

In case you are wondering how to create a custom response file, then it is pretty simple.To create response file for custom Installation

./runInstaller -record -destinationFile /oracle/ora10g/work/custom10g.rsp

Select the custom installation and choose the components which you wish to install.
When Oracle Universal Installer displays the Summary screen, perform one of the following actions:

 -Click Install to create the response file, then continue with the installation OR
    -Click Cancel and then Yes to create the response file but exit from Oracle Universal Installer without installing the software.

The response file is saved in the location that you specified using the -destinationFile option.

You Don’t need SysDBA privilege to run Awrrpt..

Well I tried to Sensationalize this post by using such a heading 🙂 Though I just wanted to point out that Awrrpt script can be run without sysdba privilege too. Actually if you open awrrpti.sql script , it contains following line

Rem    NOTES
Rem      Run as SYSDBA.  Generally this script should be invoked by awrrpt,
Rem      unless you want to pick a database other than the default.

In actual you need only two privileges to run the script. One is SELECT_CATALOG_ROLE and other is execute permission on dbms_workload_repository procedure. If you don’t give privilege explicitly on this package you get following errors

select output from table(dbms_workload_repository.awr_report_text( :dbid,
                         *
ERROR at line 1:
ORA-00904: : invalid identifier

Therefore following command’s does the trick.

grant select_catalog_role to amit;
grant execute on dbms_workload_repository to amit;

Grid Control Fails to Start

The link for Grid Control was not Working and it failed to show the login page. In this situation  checked the following things:

1. The repository database was up and running
2. The listener was up.
3. OMS server was down.

Tried to stratup the OMS server using opmnctl startall command, but it showed following error in ons.log file:

09/04/08 06:06:18 [4] ONS server initiated
09/04/08 06:06:18 [2] BIND (Address already in use)
09/04/08 06:06:18 [2] 127.0.0.1:6104 - listener BIND failed
09/04/08 06:06:18 [4] Listener thread 98311: 127.0.0.1:6104 (0x442)  terminating
09/04/08 06:06:18 [1] Local listener terminated

As it showed that the port is busy, now the obvious choice is to check which application is still holding the port. It can be done by using command:

netstat -a|grep <port_number>

i.e netstat -a |grep 6104

and the result was, no application was using this port.

But still this port was shown busy to OMS server, hence failed startup of OMS.

Then I changed the port to a different unused port in opmn.xml file and started the OMS server, it got started with new port number.

After that I stopped the OMS server and again changed the port number back to the original port number which was giving error earlier.

This time OMS started with the  old port number as well.

Cheers!!!!!

Saurabh Sood