The first problem reported by my EM 12c is an ORA-7445 error. After checking the alert log following is the exact message:
Sun Feb 19 12:09:41 2012
Errors in file /u01/app/oracle/diag/rdbms/pkldb/pkldb/trace/pkldb_j035_14712.trc (incident=52961):
ORA-07445: exception encountered: core dump [kokscold()+849] [SIGSEGV] [ADDR:0x50] [PC:0x115E16F] [Address not mapped to object] []
Incident details in: /u01/app/oracle/diag/rdbms/pkldb/pkldb/incident/incdir_52961/pkldb_j035_14712_i52961.trc
From the incident trace file:
*** 2012-02-19 12:09:43.016
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x3, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=azmq8z48r7wk4) -----
/* SQL Analyze(117,1) */ MERGE INTO MGMT_ARU_PATCH_RECOMMENDATIONS p USING (SELECT :1 as patch_guid, :2 as PATCH_TYPE, :3 as BUG_ID, :4 as REQUEST_ID, :5 as PLATFORM_ID, :6 as PRODUCT_ID, :7 as RELEASE_ID , :8 as LANGUAGE_ID, :9 as CLASSIFICATION_ID, :10 as PLATFORM, :11 as PRODUCT, :12 as RELEASE, :13 as LANGUAGE, :14 as CLASSIFICATION, :15 as ARU_TARGET_TYPE, :16 as ABSTRACT, :17 as RELEASE_DATE, :18 as LOAD_TIMESTAMP, :19 as TEXT_CONTENT, :20 as SUPPORT_LEVEL, :21 as SUPPORT_LEVEL_ID FROM DUAL) n ON (p.patch_guid = n.patch_guid) WHEN MATCHED THEN UPDATE SET p.PATCH_TYPE = n.PATCH_TYPE, p.BUG_ID = n.BUG_ID, p.REQUEST_ID = n.REQUEST_ID, p.PLATFORM_ID = n.PLATFORM_ID, p.PRODUCT_ID = n.PRODUCT_ID, p.RELEASE_ID = n.RELEASE_ID, p.LANGUAGE_ID = n.LANGUAGE_ID, p.CLASSIFICATION_ID = n.CLASSIFICATION_ID, p.PLATFORM = n.PLATFORM, p.PRODUCT = n.PRODUCT, p.RELEASE = n.RELEASE, p.LANGUAGE = n.LANGUAGE, p.CLASSIFICATION = n.CLASSIFICATION, p.ARU_TARGET_TYPE = n.ARU_TARGET_TYPE, p.ABSTRACT = n.ABSTRACT, p.RELEASE_DATE = n.RELEASE_DATE,p.LOAD_TIMESTAMP = n.LOAD_TIMESTAMP, p.TEXT_CONTENT = n.TEXT_CONTENT, p.SUPPORT_LEVEL = n.SUPPORT_LEVEL, p.SUPPORT_LEVEL_ID = n.SUPPORT_LEVEL_ID WHEN NOT MATCHED THEN INSERT (PATCH_GUID, PATCH_TYPE, BUG_ID, REQUEST_ID, PLATFORM_ID, PRODUCT_ID, RELEASE_ID, LANGUAGE_ID, CLASSIFICATION_ID, PLATFORM, PRODUCT, RELEASE, LANGUAGE, CLASSIFICATION, ARU_TARGET_TYPE, ABSTRACT, RELEASE_DATE,LOAD_TIMESTAMP, TEXT_CONTENT, SUPPORT_LEVEL, SUPPORT_LEVEL_ID) VALUES (n.PATCH_GUID, n.PATCH_TYPE, n.BUG_ID, n.REQUEST_ID, n.PLATFORM_ID, n.PRODUCT_ID, n.RELEASE_ID, n.LANGUAGE_ID, n.CLASSIFICATION_ID, n.PLATFORM, n.PRODUCT, n.RELEASE, n.LANGUAGE, n.CLASSIFICATION, n.ARU_TARGET_TYPE, n.ABSTRACT, n.RELEASE_DATE,n.LOAD_TIMESTAMP, n.TEXT_CONTENT, n.SUPPORT_LEVEL, n.SUPPORT_LEVEL_ID)
----- PL/SQL Call Stack -----
object line object
handle number name
0x80bd47c0 11774 package body SYS.DBMS_SQLTUNE_INTERNAL
0x80b50a80 7 SYS.WRI$_ADV_SQLTUNE
0x82d550f8 545 package body SYS.PRVT_ADVISOR
0x82d550f8 2613 package body SYS.PRVT_ADVISOR
0x80bf0da8 241 package body SYS.DBMS_ADVISOR
0x81987268 772 package body SYS.DBMS_SQLTUNE
0x8187d290 4 anonymous block
*** 2012-02-19 12:09:42.839
> (0x115e16f) mov 0x50(%r11),%rdx
(0x115e173) movzbl 0x1(%r13),%ecx
(0x115e178) movzbl 0x1(%r15),%r9d
(0x115e17d) mov 0x8(%rdx),%edx
(0x115e180) lea 0x10(%r13),%r8
The error seems to be coming from some automated scheduled SQL Tuning Task (Yet to figureout that). Searching at MOS gives reference to bug 9594372, which shows that the problem is with merge statement. It says that applying the patch will fix the issue.
While checking the error from my EM12c, it also gives a search button to search the error at MOS( I have already installed the MOS plug-in) which I found is very good thing as I need not to login to separate MOS windows.
Oracle provides dbms_scheduler.copy_job function to copy oracle scheduler jobs which works great. But problem is that it cannot copy jobs which have arguments.
Need to copy jobs can arise due to following scenarios
1)Runaway jobs – When Scheduler job is running without any SID. This means OS session is not present and this will not clear. These jobs have to be recreated to resolve the issue.
2)Jobs can not be stopped – many times jobs cannot be stopped by dbms_scheduler.stop_job(<jobname>,true) as they error out with “Internal error”
3)We need to have similar job but under different schema or with different name
I have written below code which would prompt for username and jobname and will copy the job along with its arguments
declare
l_owner varchar2(30) :='&job_owner';
l_old_job varchar2(30) :='&job_name';
l_new_job varchar2(30);
cnt integer;
cursor c_arg is select argument_position,value from DBA_SCHEDULER_JOB_ARGS where job_name = l_old_job and owner = l_owner order by argument_position;
begin
l_new_job :='INTERMEDIATE_CPY';
select count(*) into cnt from dba_scheduler_jobs where job_name = l_new_job and owner = l_owner;
if ( cnt > 0 ) then
dbms_output.put_line ('dropping scheduler job INTERMEDIATE_CPY');
dbms_scheduler.drop_job(l_owner||'.'||l_new_job,true);
end if;
sys.dbms_scheduler.copy_job(l_owner||'.'||l_old_job,l_owner||'.'||l_new_job);
sys.dbms_scheduler.drop_job(l_owner||'.'||l_old_job,true);
sys.dbms_scheduler.copy_job(l_owner||'.'||l_new_job,l_owner||'.'||l_old_job);
for v_arg in c_arg
loop
dbms_output.put_line('Setting ARGUMENT_POSITION '||v_arg.argument_position||' to value '||v_arg.value);
sys.dbms_scheduler.set_job_argument_value(job_name =>l_owner||'.'||l_old_job,ARGUMENT_POSITION=>v_arg.argument_position,ARGUMENT_VALUE =>v_arg.value);
end loop;
sys.dbms_scheduler.enable(l_owner||'.'||l_old_job);
sys.dbms_scheduler.drop_job(l_owner||'.'||l_new_job,true);
end;
/
In case you don’t copy the arguments, jobs will fail with status as STOPPED and give REASON as “Job slave process was terminated”. Also it can generate following ORA-07445 error
ORA-07445: exception encountered: core dump [kpubsuuc()+197] [SIGSEGV] [Address not mapped to object] [0x000000000] [] []
In this post we will discuss the Automated discovery of Targets in Enterprise Manager Cloud Control(EM 12c). Once you have installed agents, you can schedule automatic discovery of new targets. Let’s get started
1)Click on Add Target>Configure Auto Discovery from the Setup menu
2)Click the Configure icon in the Multiple Target-type Discovery on Single Host row in the Discovery table.
‘
3)Select the host in the table and click Configure.
4)Set the schedule at which the discovery job will be run, in days. This schedule will be applied to all selected hosts. By default the job will run every 24 hours.
Select the Oracle component types you want to search the host for and click Ok.
5)Click “Run Discovery now” and it will report the discovered targets.
6) Next is to configure the Targets. We will first start with oradbdev-clu i.e RAC cluster
We have added all the hosts part of cluster. In case management agent is not installed on all hosts , they will not appear in “Available hosts” list
7)Next we select Cluster database tintin and click on configure
8)You will be required to put dbsnmp password for configuring database montioring. By default dbsnmp account is locked. You can verify using following sql
select username,account_status from dba_users where username='DBSNMP';
USERNAME ACCOUNT_STATUS
------------------------------ --------------------------------
DBSNMP EXPIRED & LOCKED
Unlock and reset password as below
sql>alter user DBSNMP account unlock identified by oracle;
Do test connection and verify all information is correct.
9)Then configure ASM instances and you will be required to enter ASM password
10)We are done now. We can go to Targets ->Databases ->Tintin to view the database. It would look like below
This Post would describe how to install Oracle Management Agent 12c in silent mode. We need to download the agent software which can be done using emcli utility which is present on OMS host.Login to OMS host and execute emcli utility as mentioned below
[oracle@oradbdev04]~/Middleware/oms/bin% emcli login -username=sysman
Enter password
Login successful
Following command gets list of agent software present on grid control
[oracle@oradbdev04]~/Middleware/oms/bin% emcli get_supported_platforms
Getting list of platforms ...
Check the logs at /home/oracle/Middleware/oms/bin/agent.log
About to access self-update code path to retrieve the platforms list..
Getting Platforms list ...
-----------------------------------------------
Version = 12.1.0.1.0
Platform = Linux x86-64
-----------------------------------------------
Platforms list displayed successfully.
Since this is a new Grid installation on Linux x86-64 we get only 1 agent software. If you need to install agent on different platform say Solaris, then you need to download software using self update option.
Note that you will have to pass your My oracle support credentials while configuring self-update.
Since our target host is also Linux x86-64 we can proceed. In below example we have specified that our 12.1.0.1.0 agent software should be placed in /tmp for Linux x86-64 platform
[oracle@oradbdev04]~/Middleware/oms/bin% ./emcli get_agentimage -destination=/tmp -platform="Linux x86-64" -version=12.1.0.1.0
Platform:Linux x86-64
Destination:/tmp
=== Partition Detail ===
Space free : 3 GB
Space required : 1 GB
Check the logs at /tmp/get_agentimage_2012-02-04_11-56-01-AM.log
Setting property ORACLE_HOME to:/home/oracle/Middleware/oms
calling pulloneoffs with arguments:/home/oracle/Middleware/oms/home/oracle/Middleware/oms/sysman/agent/12.1.0.1.0_AgentCore_226.zip12.1.0.1.0linux_x64
Check this logs for more information: /home/oracle/Middleware/oms/sysman/prov/agentpush/logs
[oracle@oradbdev04]~/Middleware/oms/bin% ls -l /tmp/12.1.0.1.0_AgentCore_226.zip
-rw-r----- 1 oracle dba 235679029 Feb 4 11:56 /tmp/12.1.0.1.0_AgentCore_226.zip
You should copy this file to the target host and unzip it.
To perform installation , we need to use agentDeploy.sh script. You can either use agent.rsp in unzipped directory or pass parameters at command line
AGENT_BASE_DIR ->Location where Agent software will be installed OMS_HOST –>OMS host name EM_UPLOAD_PORT –>Agent upload port. Script will try both http/https port. AGENT_REGISTRATION_PASSWORD –>Password for secure registration with OMS
cd $AGENT_HOME/core/12.1.0.1.0/bin ./emctl status agent ./emctl upload agent
$ ./emctl upload agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully
Next step is to discover the targets. You can either manually add them or use automated discovery of new Oracle targets on machines which already have agent installed. Refer to EM12c:Automated discovery of Targets for automatic discovery and target configuration.
In this blog post I will be discussing listener startup issues faced in 11gR2 RAC. I will be constantly updating this post based on my experiences or any comments on this blog post.
Let’s get started.You will experience following errors while starting listener using srvctl
[oracle@prod01]~% srvctl start listener -n prod01
PRCR-1013 : Failed to start resource ora.LISTENER.lsnr
PRCR-1064 : Failed to start resource ora.LISTENER.lsnr on node prod01
CRS-5016: Process "/oragrid/product/11.2.0.2/bin/lsnrctl" spawned by agent "/oragrid/product/11.2.0.2/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/home/oragrid/product/11.2.0.2/log/prod01/agent/crsd/oraagent_oracle/oraagent_oracle.log"
CRS-5016: Process "/oragrid/product/11.2.0.2/bin/lsnrctl" spawned by agent "/oragrid/product/11.2.0.2/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/home/oragrid/product/11.2.0.2/log/prod01/agent/crsd/oraagent_oracle/oraagent_oracle.log"
CRS-2674: Start of 'ora.LISTENER.lsnr' on 'prod01' failed
Issue 1: – Incorrect ORACLE_HOME entry in listener.ora
This issue can be verified by attempting to start listener by lsnrctl utility. Please note that you need to use $GRID_HOME/bin/lsnrctl utility to manage listener in 11gR2 RAC
[oracle@prod01]~% lsnrctl start
LSNRCTL for Linux: Version 11.2.0.2.0 - Production on 31-JAN-2012 08:09:52
Copyright (c) 1991, 2010, Oracle. All rights reserved.
Starting /oragrid/product/11.2.0.2/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 11.2.0.2.0 - Production
System parameter file is /oragrid/product/11.2.0.2/network/admin/listener.ora
Log messages written to /oracle/diag/tnslsnr/prod01/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.160)(PORT=1521)))
TNS-01201: Listener cannot find executable /oracle/product/11.2/bin/oracle for SID orcl01
Listener failed to start. See the error message(s) above..
Above error indicates that it is not able to find ‘oracle’ executable in specified path. But this is not our correct ORACLE_HOME. Checking listener.ora we found that this home is coming from ORACLE_HOME variable
Correct ORACLE_HOME is oracle/product/11.2.0.2 which needs to be updated correctly in listener.ora. After adding we were able to start the listener.
Issue 2: CRS resource ora.[node_name].ons is down
While debugging listener startup issue, you found that resource ora.[node_name].ons is not starting. Listener (including SCAN_LISTENER) is dependent on ora.ons resource
Checking $GRID_HOME/opmn/logs/ons.log[node_name], we see following messages
12/01/12 05:32:48 [ons-listener] Could not get address information for localhost 6100.
12/01/12 05:32:49 [internal] getaddrinfo(localhost, 6100, 1) failed (Name or service not known):
12/01/12 05:33:05 [internal] getaddrinfo(localhost, 6100, 1) failed (Name or service not known):
12/01/12 05:33:15 [internal] getaddrinfo(localhost, 6100, 1) failed (Name or service not known):
12/01/12 05:33:15 [internal] getaddrinfo(localhost, 6100, 1) failed (Name or service not known):
12/01/12 05:33:15 [internal] getaddrinfo(localhost, 6100, 1) failed (Name or service not known):
Issue is that your host is not able to resolve localhost setting.You can verify this by issuing ping localhost command.This issue can be resolved by adding following entry in /etc/hosts or to DNS
127.0.0.1 localhost
Issue 3: VIP and ora.[node_name].ons are not starting on one node
In this issue check that Bcast and Mask settings for Public interface are same on all nodes.
This post describes the steps to upgrade an existing 11g Enterprise Manager Grid Control (11.1.0.1.0) to 12c (12.1.0.1.0). The Complete Upgrade process will be carried out under 3 different stages:
1. Patch existing 11g OMS to get the 12c upgrade console. 2. Deploy and Configure 12c agents. 3. Upgrade 11g OMS to 12c.
The approach used for this upgrade is 1-system upgrade approach. This approach enables you to upgrade to Enterprise Manager Cloud Control on the same host where your earlier release of Enterprise Manager is running. This approach also upgrades the Oracle Management Repository (Management Repository) in the existing database. Since the upgrade happens on the same host, there is a reasonable downtime involved. This approach does not refer to upgrading of an Enterprise Manager system in an environment with one Oracle Management Service (OMS). It refers to upgrading of an Enterprise Manager system on the same host as the old one, and having only one Enterprise Manager system existing at any point.
The supported earlier releases for upgrade include:
Oracle Management Agent 10g Release 2 (10.2.x.x.x), Oracle Management Agent 11g Release 1 (11.1.0.1.0), Oracle Management Service 10g Release 5 (10.2.0.5.0) and Oracle Management Service 11g Release 1(11.1.0.1.0).
Unlike earlier releases Oracle Management Service 12c communicates only with Oracle Management Agent 12c. Therefore, unlike the earlier releases, you must first upgrade the Management Agents in your system before upgrading your existing OMS.
1. Patch existing 11g OMS to get the 12c upgrade console
To start the upgrade process we need to get the 12c upgrade console, which we can get by installing patch for bug# 10044087 on your existing Enterprise Manager system. To apply this patch OMS needs to be down which means that there is a downtime required for the existing GridControl environment. Download the patch and apply for your platform click HERE for download link.
This patch application is a simple patch apply process with opatch. Following are the mandatory things before you start patch apply:
— It is mandatory to apply PSU1(10065631) or later on the existing OMS home
— There is a chance of hitting bug 9953045 and bug 12360281, to escape these situations, set SQLNET.ENCRYPTION_TYPES_SERVER= (3DES168) in sqlnet.ora of your oracle database sqlnet file.
Follow the patch readme to apply this patch. Once this patch is applied and the post scripts(puc_dblink_pkgdef.sql, puc_dblink_pkgbody.sql and pre_upg_console.sql) are run and OMS is started, you can see the “Enterprise manager 12c Upgrade Console” under “Deployments” tab of Grid Control.
2. Deploy and Configure 12c agents:
Download the following software, and stage them to an accessible location:
– Oracle Management Agent 12c – All the required plug-ins
Don’t unzip the binaries after downloading them. Verify the checksum after downloading the zip files. To manage information about the location of the core Management Agent software and its associated plug-ins, follow these steps:
1. In Grid Control, click Deployments. 2. On the Deployments page, in the Upgrade section, click Enterprise Manager 12c Upgrade Console. 3. On the Upgrade Console page, in the Select Upgrade Type section, select one of the following:
– To upgrade your Enterprise Manager system with downtime (even if you manually install Management Agents), select 1-System. – To upgrade your Enterprise Manager system with “near zero” downtime (even if you manually install Management Agents), select 2-System. – To upgrade your Enterprise Manager system with downtime on a different host, select 1-System on a Different Host.
Select 1-system approach here.
Enterprise Manager Grid Control refreshes the page and displays a table with a list of tasks you need to perform for the upgrade approach you selected.
4. In the Preupgrade Steps section, from the table, click Manage Software. 5. On the Manage Software page, in the Agent Upgradability and Target Upgradability pie charts, click the hyperlinks in the respective legends and identify the Management Agents and targets that can be upgraded, and that cannot be upgraded due to unavailability of the Management Agent software or the plug-in software.
6. In the Provide Software Location section, enter the absolute path to the directory where the core Management Agent software and the plug-in software are present for the required platforms.
For example, /u01/app/plugin
And then click Validate to register that location with the Enterprise Manager system. Once validate is complete you will see the below pie charts adn these should be in all green. If they are not in green, it means that you have missed some plugin(you will find the list of all required plugins at the botton of same page), download and stage the missing plugin and validate again.
Validate
Once the software have been verified we can continue to deploy and configure the software binaries of Oracle Management Agent 12c.
1. In Grid Control, click Deployments. 2. On the Deployments page, in the Upgrade section, click Enterprise Manager 12c Upgrade Console. 3. On the Upgrade Console page, in the Select Upgrade Type section, select 1-system approach.
In the “Preupgrade Steps” section, from the table, click Deploy and Configure Agents.
4. On the Deploy and Configure Agents page, for Operation Name, enter a unique name for the deployment operation you are about to perform. The custom name you enter can be any intuitive name, and need not necessarily be in the same format as the default name. For example, Deploy_Agents_Phase1_2010_12_27. 6. In the Select Operation Type section, select Deploy Agent and Plug-In Software and Configure Agent and Plug-In Software. 7. In the Search Agents section, search and add the existing, old Management Agents for which you want to deploy the software. In the table that lists the Management Agents, enter an installation base directory and an instance home directory for each of the Management Agents.
In the Agent Credentials section, retain the default selection, that is, Use Oracle Home Preferred Credentials,if you have already set these, so that the preferred credentials stored in the Management Repository can be used for this job.In the OMS Host and Port section, validate the name and the secure port of the host where you plan to install Oracle Management Service 12c. To change the values, click Edit. After filling up the port details click next, do not select any pre and post upgrade script. Provide the root credentials and click submit. Agent will start a job with the name that you have provided, click at the job to see the status.
Generate Health Report of Deployed Oracle Management Agents. Follow the steps by clicking the “Generate help report” , once it is completed, you will see following screen:
Verify and sign off the health check report. After this you have to Switch-Over to 12c agents. Under “Agent Upgrade Steps” click switch agents:
Search for all the agents and provide the oracle preferred credentials. Click submit.
Once this job completes, the old agents will be stopped and new agents from 12g home will be started.
3. Upgrade 11g OMS to 12c
Download and unzip the 12c software from the link mentioned earlier in the post. It is mandatory to meet all the required prerequisites mentioned at ORACLE DOCUMENTATION otherwise the installation will fail at some later stage. I will list one import prerequisite step here:
Copy EMkey to repository: To do so, check the status of emkey as: [oracle@cloud1 bin]$ emctl status emkey
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password : xxxxx
The EMKey is configured properly.
Copy the emkey as:
$ emctl config emkey -copy_to_repos
After meeting all the prerequisites, continue with the OMS upgrade:
Invoke runInstaller from 12c unzip location:
$ ./runInstaller
Choose your option to receive updates via MOS and click next. I opted to remain uninformed from MOS updates.
On the Software Updates screen, select one of the following sources from where the software updates can be installed while the Enterprise Manager system gets upgraded. If you do not want to apply them now, then select Skip. I opted to skip these updates for the installation. Click next.
On the Prerequisite Checks screen, check the status of the prerequisite checks run by the installation wizard, and verify whether your environment meets all the minimum requirements for a successful upgrade. It may show “Failed” for some of the packages, make sure that those were installed or higher versions of those packages are already in place. Once confirmed all settings, click next.
Under Install Type, select to upgrade an existing instance option and choose 1-system upgrade approach. It will automatically find the existing middleware home. Enter a valid middleware home location where it will install the required components for 12c. Provide the location of Middleware home . Click next.
Provide sys and sysman password and click next.
You can select the plug-ins that you want to install, the already installed plugins will be greyed-out. Click next after selecting the plugins.
If you are upgrading from Enterprise Manager 11g Grid Control Release 1 (11.1.0.1.0), then on the Extend WebLogic Server Domain screen, validate the AdminServer host name and its port, and the WebLogic user name. Enter the WebLogic user account password for extending the existing Oracle WebLogic Server Domain to the upgraded release.
Installation status.. once the installation finishes, it will ask you to run allroot.sh as root user. Run the script as root user and click finish.
Launch the Cloud Console as you will see like:
Login using sysman and now you are ready to control clouds 🙂
This blog reflect our own views and do not necessarily represent the views of our current or previous employers.
The contents of this blog are from our experience, you may use at your own risk, however you are strongly advised to cross reference with Product documentation and test before deploying to production environments.
Recent Comments