GRID

How To Configure Exadata Database Machine in Enterprise Manager Cloud Control 13c (OEM13c)

I have followed the steps in Oracle Documentation link: https://docs.oracle.com/cd/E63000_01/EMXIG/ch2_deployment.htm#EMXIG215 to configure Exadata Database Machine in OEM13c. If you want to configure your Exadata in OEM13c you have to follow the above mentioned link.
In this post I will share the mandatory steps for configuration, and some of the issues which I faced while configuring the Exadata on OEM13c.
NOTE: OEM13c agents only needs to be deployed on compute nodes on your Exadata Machine.

Step 1: Deploy Exadata Plug-In in OEM13c.

Step 2: For an EM agent to communicate with ILOM SP, there must be a user created on ILOM SP, on all the Compute Nodes.
Create A Database Server ILOM SP (Service Processor) User.
Login to Compute Node ILOM with “root” user”
# cd /SP/users
# create oemuser
Creating user…
Enter new password: ********
Enter new password again: ********

Created /SP/users/oemuser

Change to the new user’s directory and set the role:

# cd oemuser
/SP/users/oemuser

set role=’cro’
Set ‘role’ to ‘cro’

Now test the ILOM user ID created:

For Exadata X5-2:
# ipmitool -I lanplus -H <ComputeNodeILOMHostname> -U oemuser -P xxxxxx -L USER sel list last 10
It should display some results.

Now run the above steps on all Compute Node ILOMs.

STEP 3: Push the OEM agent to Compute nodes.
From OEM13c console, select Setup from top right corner, and then Add Target, and the Add Target Manually. Put the Compute Node’s hostname, and then select your OS version. Fill-in the rest of the details on the screen and click Deploy.  It will Deploy the agent on the Compute Nodes you have mentioned.

 

Step 4: Run discovery Precheck Script:
To ensure that discovery of Exadata Machine complete without any issues, you need to run exadataDiscoveryPreCheck.pl. This script is available under OEM13c OMS server Exadata plug-in location i.e:
<OMS_agent installation directory>/plugins/oracle.sysman.xa.discovery.plugin_12.1.0.3.0/discover/dbmPreReqCheck/exadataDiscoveryPreCheck.pl, verify the path as per your configuration and run the script. You can also download the script from MOS Note: 1473912.1.

NOTE: For Infiniband user you have to use “nm2user” and its default password is changeme.

This script showed following errors to me:
Verifying setup files consistency... 
------------------------------------ 
Verifying cell nodes... 
Cell node <CellNode Name> is missing in one of the setup files. 
Cell node <CellNode Name>.domain is missing in one of the setup files. 
Cell node <CellNode Name>.domain is missing in one of the setup files. 
Cell node <CellNode Name> is missing in one of the setup files. 
Cell node <CellNode Name> is missing in one of the setup files. 
Cell node <CellNode Name>.domain is missing in one of the setup files. 
Verifying infiniband nodes... 
Infiniband node <IBNode Name>.domain is missing in one of the setup files. 
Infiniband node <IBNode Name> is missing in one of the setup files. 
Infiniband node <IBNode Name>.domain is missing in one of the setup files. 
Infiniband node <IBNode Name> is missing in one of the setup files. 
Infiniband node null is missing in one of the setup files. 
Verifying KVM nodes... 
KVM node null is missing in one of the setup files. 
Verifying PDU nodes... 
PDU node <PDUNode Name> is missing in one of the setup files. 
PDU node <PDUNode Name> is missing in one of the setup files. 
PDU node <PDUNode Name>.domain is missing in one of the setup files. 
PDU node <PDUNode Name>.domain is missing in one of the setup files. 
Setup files are not consistent ===> Not ok 
* Please make sure that node information in both parameter and schematic files 
is consistent. 
======================================================= 
* Please make sure ciphers are correctly set in all cell and compute nodes. 
Verifying SSH cipher definition for <CellNode Name> cell node... 
None of the expected ciphers were found in sshd_config file ===> Not ok 
* Please make sure ciphers are correctly set in sshd_config file. 
== =========================================================

 So there were two issues:
1. Parameter file and Schematic file were not in sync with each other.
2. Missing valid cipher in cellnodes’ sshd_config file.
For parameter files issue, we need to check two files under /opt/oracle.SupportTools/onecommand, em.params and databasemachine.xml, and have to make sure that entries are same
in these files. In my case all the names under em.params were with fqdn and under databasemachine.xml these were without fqdn. I modified em.params to remove the fqdn from
all names.
For cipher issue, as the compute nodes did not error out for valid ciphers, I have copied one cipher entry from Compute Node to all the Cell Nodes and restarted the sshd service.

After making these two changes I ran exadataDiscoveryPreCheck.pl script again and it came out clean.

STEP 5: Discovering an Exadata Database Machine

1. From the Enterprise Manager home page, select the Setup menu (upper right corner), Add Target, and then Add Targets Manually.

2. On the Add Targets Manually page, click Add Targets Using Guided Process. From Add Using Guided Process window, select Oracle Exadata Database Machine from the list and click Add.

3. On the Oracle Exadata Database Machine Discovery page, select one of the following tasks:
13c target type
12c target type
I opted for 13c target type.

4. On the Discovery Inputs page, enter the following information
For the Discovery Agents section:
Agent URL: The Agent deployed on compute node. Click the search icon to select from available URLs.
For the Schematic Files section:
Once you have specified the Agent URL, a new row (hostname and schematic file information) is automatically added. The default schematic file, databasemachine.xml, describes the hardware components of the Exadata Database Machine.
Click Set Credential to set the credentials for the host.
Check/modify the schematic file location.
Select the schematic file name from drop-down menu.

5. On the InfiniBand Discovery page, enter the following information:
IB Switch Host Name: The InfiniBand switch host name. The IB Switch host name is usually pre-populated.
InfiniBand Switch ILOM host credential: The user name (usually ilom-admin or ilom-operator) and password for the InfiniBand switch ILOM host.

Rest of the steps are self explanatory and can be filled easily.

On Credentials page, after filling root password, you will get two options under SNMP credentials:
— Credential Type SNMPV1
— Credential Type SNMPV3
I opted for SNMPV3 and it requires EXACLI username/password. So you have to create ExaCli user as described at
http://docs.oracle.com/cd/E50790_01/doc/doc.121/e50471.pdf on Page 384.
Create the Exacli user and provide the information asked under SNMPV3.

Click Submit and it will take some time to discover the Exadata DB Machine.

After this, you can see “Exadata” under Targets tab on OEM13c home page.

 

Upgrade Enterprise Manager Cloud Control 12.1.0.1 to 12.1.0.2

Oracle released 12.1.0.2 Enterprise Manager few days back and have released it for most of platforms.

You can directly upgrade Enterprise Manager Cloud Control 12c Release 1 (12.1.0.1) [with or without Bundle Patch 1] to 12.1.0.2
There are no pre patches required for this. But this upgrade requires considerable downtime. Oracle has termed this upgrade as 1-system upgrade approach which means that upgrades your Enterprise Manager Cloud Control on the same host—upgrades Oracle Management Service (OMS) on the same host and Oracle Management Repository (Management Repository) in the existing database.

I decided to upgrade my test EM 12c setup running on RHEL 5.6 x86_64 bit with single OMS and 11.2.0.3 database.Whole upgrade went fine without any major issues. To get started you need to download the software from here

I will be upgrading 12.1.0.1 running on Linux x86_64 bit. You can download the software from
http://download.oracle.com/otn/linux/oem/121020/em12cr2_linux64_disk1.zip
http://download.oracle.com/otn/linux/oem/121020/em12cr2_linux64_disk2.zip
http://download.oracle.com/otn/linux/oem/121020/em12cr2_linux64_disk3.zip

Once all files are downloaded, unzip them in single directory location.
To start the installation , run the runInstaller

./runInstaller

First screen asks for MOS details. You can choose to ignore it by unchecking box and click next.This will generate warning, press ok and click next

Next screen is related to software updates. You can “search for updates” and then select “My Oracle Support” to download the updates.We opted to “skip” this part.

On pre-requisite screen, installer checks whether your environment meets all the minimum requirements for a successful upgrade.
If some checks result in Warning or Failed status, then investigate and correct the problems before you proceed with the upgrade.

On the Installation Types screen, select Upgrade an Existing Enterprise Manager System, then select One System Upgrade. Then, select the OMS home you want to upgrade.

Next select the location for middleware home. 12.1.0.2 is out of place upgrade,so you need to specify new middleware home (without EM grid control) or a new home.

Next screen you pass connection details for EM repository.After you enter sys/sysman password it prompts you to stop oms

Stop the OMS using emctl utility.

$OMS_HOME/bin/emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down

Once OMS is stopped, we press ok on message and again click Next.Installer complains for pre-requisite check for emkey .This can be corrected by copying emkey to database repository.

Syntax for this is

$/bin/emctl config emkey -copy_to_repos_from_file -repos_host-repos_port-repos_sid-repos_user[-repos_pwd] -emkey_file/sysman/config/emkey.ora

I have replaced this with actual values for my setup

[oracle@oradbdev04]~% /home/oracle/Middleware/oms/bin/emctl config emkey -copy_to_repos_from_file -repos_host oradbdev04 -repos_port 1521 -repos_sid em12c -repos_user sysman -repos_pwd oracle123 -emkey_file /home/oracle/Middleware/oms/sysman/config/emkey.ora 
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0  
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
Enter Admin User's Password : 
The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".

Next screen asks you to disable CBO gather stats job and set job_queue_processes to 0. You can press “Yes” and installer will automatically make changes for you.

We got next screen informing about some more pre-requisites which are not mandatory. We clicked ok and skipped them

On the Plug-In Upgrade screen, it will list plugins which are currently installed and new version post upgrade. Review them and click next

On the Plug-In Deployment screen, select the optional plug-ins you want to deploy in addition to the plug-ins that will automatically be upgraded while upgrading the OMS. ( I forgot to take screenshot of this page)

On the Extend WebLogic Server Domain screen , validate Adminserver details and enter WebLogic user account password.This is required to create a new WebLogic domain (GCDomain) on the same port and host name as the AdminServer used by the earlier release of the OMS you are upgrading.

We also need to enter new location for OMS instance base directory. As per oracle docs, it can be either inside or outside middleware home. Recommended approach is to keep it inside middleware home. We decided to use /home/oracle/product/middleware/gc_inst

You are next presented with summary screen and asked to click next

Next screen shows installation progress. This took nearly 1.5 hours.

Last step is to run $OMS_HOME/allroot.sh as root user

[root@oradbdev04 oms]# ./allroot.sh
Starting to execute allroot.sh .........
Starting to execute /home/oracle/product/middleware/oms/root.sh ......
 Running Oracle 11g root.sh script...
The following environment variables are set as:
 ORACLE_OWNER= oracle
 ORACLE_HOME= /home/oracle/product/middleware/oms
Enter the full pathname of the local bin directory: [/usr/local/bin]:
 The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
 [n]:
 The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
 [n]:
 The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
 [n]:
Entries will be added to the /etc/oratab file as needed by
 Database Configuration Assistant when a database is created
 Finished running generic part of root.sh script.
 Now product-specific root actions will be performed.
 /etc exist
 /home/oracle/product/middleware/oms
 Finished execution of /home/oracle/product/middleware/oms/root.sh ......

 

Press ok on installer and we are finished with 12c installation.

Final screen lists down the url to access Enterprise manager grid control (12.1.0.2) and Weblogic Admin server.


Since one system upgrade only upgrades OMS and repository , we are still left with upgrading our central management agent.

1)Ensure that the Management Agents you want to upgrade are up and running.
2)Ensure that the Management Agents you want to upgrade are secure.

From the Setup menu, select Manage Cloud Control, then select Upgrade Agents. Click Add, select the Management Agents you want to upgrade, then click Select.

Select the Central management agent (which in this case is oradbdev04) and submit the job

Since we do not have root privileges, we have to run the $AGENT_BASE_DIR/core/12.1.0.2.0/root.sh script on the host after the upgrade.

 

You can monitor the agent install process

Note that oracle installs the agent in same base directory but creates new directory 12.1.0.2.0.

Before upgrade
/home/oracle/Middleware/agent/core/12.1.0.1.0

Post upgrade
/home/oracle/Middleware/agent/core/12.1.0.2.0

If you check agent status, you will find that oracle has automatically started the agent from new oracle home.

Run root.sh to complete the upgrade process

[root@oradbdev04 12.1.0.2.0]# ./root.sh
Finished product-specific root actions.
/etc exist
Finished product-specific root actions.

You can now upgrade the other management agent manually on the targets. Agent 12.1.0.1 is compatible with EMGC (12.1.0.2)

References

Oracle® Enterprise Manager Cloud Control Upgrade Guide12c Release 2 (12.1.0.2)

ORA-7445 core dump [kokscold()+849] from EM 12c

The first problem reported by my EM 12c is an ORA-7445 error. After checking the alert log following is the exact message:

Sun Feb 19 12:09:41 2012
Errors in file /u01/app/oracle/diag/rdbms/pkldb/pkldb/trace/pkldb_j035_14712.trc (incident=52961):
ORA-07445: exception encountered: core dump [kokscold()+849] [SIGSEGV] [ADDR:0x50] [PC:0x115E16F] [Address not mapped to object] []
Incident details in: /u01/app/oracle/diag/rdbms/pkldb/pkldb/incident/incdir_52961/pkldb_j035_14712_i52961.trc

From the incident trace file:

*** 2012-02-19 12:09:43.016
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x3, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=azmq8z48r7wk4) -----
/* SQL Analyze(117,1) */ MERGE INTO MGMT_ARU_PATCH_RECOMMENDATIONS p USING (SELECT :1  as patch_guid, :2  as PATCH_TYPE, :3  as BUG_ID, :4  as REQUEST_ID, :5  as PLATFORM_ID, :6  as PRODUCT_ID, :7  as RELEASE_ID , :8  as LANGUAGE_ID, :9  as CLASSIFICATION_ID, :10  as PLATFORM, :11  as PRODUCT, :12  as RELEASE, :13  as LANGUAGE, :14  as CLASSIFICATION, :15  as ARU_TARGET_TYPE, :16  as ABSTRACT, :17  as RELEASE_DATE, :18  as LOAD_TIMESTAMP, :19  as TEXT_CONTENT, :20  as SUPPORT_LEVEL, :21  as SUPPORT_LEVEL_ID FROM DUAL) n ON (p.patch_guid = n.patch_guid) WHEN MATCHED THEN UPDATE SET p.PATCH_TYPE = n.PATCH_TYPE, p.BUG_ID = n.BUG_ID, p.REQUEST_ID = n.REQUEST_ID, p.PLATFORM_ID = n.PLATFORM_ID, p.PRODUCT_ID = n.PRODUCT_ID, p.RELEASE_ID = n.RELEASE_ID, p.LANGUAGE_ID = n.LANGUAGE_ID, p.CLASSIFICATION_ID  = n.CLASSIFICATION_ID, p.PLATFORM = n.PLATFORM, p.PRODUCT = n.PRODUCT, p.RELEASE = n.RELEASE, p.LANGUAGE = n.LANGUAGE, p.CLASSIFICATION = n.CLASSIFICATION, p.ARU_TARGET_TYPE = n.ARU_TARGET_TYPE, p.ABSTRACT = n.ABSTRACT, p.RELEASE_DATE = n.RELEASE_DATE,p.LOAD_TIMESTAMP = n.LOAD_TIMESTAMP, p.TEXT_CONTENT = n.TEXT_CONTENT, p.SUPPORT_LEVEL = n.SUPPORT_LEVEL, p.SUPPORT_LEVEL_ID = n.SUPPORT_LEVEL_ID WHEN NOT MATCHED THEN  INSERT (PATCH_GUID, PATCH_TYPE, BUG_ID, REQUEST_ID, PLATFORM_ID, PRODUCT_ID, RELEASE_ID, LANGUAGE_ID, CLASSIFICATION_ID, PLATFORM, PRODUCT, RELEASE, LANGUAGE, CLASSIFICATION, ARU_TARGET_TYPE, ABSTRACT, RELEASE_DATE,LOAD_TIMESTAMP, TEXT_CONTENT, SUPPORT_LEVEL, SUPPORT_LEVEL_ID) VALUES (n.PATCH_GUID, n.PATCH_TYPE, n.BUG_ID, n.REQUEST_ID, n.PLATFORM_ID, n.PRODUCT_ID, n.RELEASE_ID, n.LANGUAGE_ID, n.CLASSIFICATION_ID, n.PLATFORM, n.PRODUCT, n.RELEASE, n.LANGUAGE, n.CLASSIFICATION, n.ARU_TARGET_TYPE, n.ABSTRACT, n.RELEASE_DATE,n.LOAD_TIMESTAMP, n.TEXT_CONTENT, n.SUPPORT_LEVEL, n.SUPPORT_LEVEL_ID)


----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0x80bd47c0     11774  package body SYS.DBMS_SQLTUNE_INTERNAL
0x80b50a80         7  SYS.WRI$_ADV_SQLTUNE
0x82d550f8       545  package body SYS.PRVT_ADVISOR
0x82d550f8      2613  package body SYS.PRVT_ADVISOR
0x80bf0da8       241  package body SYS.DBMS_ADVISOR
0x81987268       772  package body SYS.DBMS_SQLTUNE
0x8187d290         4  anonymous block

*** 2012-02-19 12:09:42.839
> (0x115e16f) mov 0x50(%r11),%rdx
  (0x115e173) movzbl 0x1(%r13),%ecx
  (0x115e178) movzbl 0x1(%r15),%r9d
  (0x115e17d) mov 0x8(%rdx),%edx
  (0x115e180) lea 0x10(%r13),%r8

The error seems to be coming from some automated scheduled SQL Tuning Task (Yet to figureout that). Searching at MOS gives reference to bug 9594372, which shows that the problem is with merge statement. It says that applying the patch will fix the issue.

While checking the error from my EM12c, it also gives a search button to search the error at MOS( I have already installed the MOS plug-in) which I found is very good thing as I need not to login to separate MOS windows.

EM12c:Automated discovery of Targets

In this post we will discuss the Automated discovery of Targets in Enterprise Manager Cloud Control(EM 12c). Once you have installed agents, you can schedule automatic discovery of new targets. Let’s get started

1)Click on Add Target>Configure Auto Discovery from the Setup menu

2)Click the Configure icon in the Multiple Target-type Discovery on Single Host row in the Discovery table.

3)Select the host in the table and click Configure.

4)Set the schedule at which the discovery job will be run, in days. This schedule will be applied to all selected hosts. By default the job will run every 24 hours.

Select the Oracle component types you want to search the host for and click Ok.

5)Click “Run Discovery now” and it will report the discovered targets.

6) Next is to configure the Targets. We will first start with oradbdev-clu i.e RAC cluster

We have added all the hosts part of cluster. In case management agent is not installed on all hosts , they will not appear in “Available hosts” list

7)Next we select Cluster database tintin and click on configure

8)You will be required to put dbsnmp password for configuring database montioring. By default dbsnmp account is locked. You can verify using following sql

select username,account_status from dba_users where username='DBSNMP';
USERNAME ACCOUNT_STATUS
------------------------------ --------------------------------
DBSNMP EXPIRED & LOCKED

Unlock and reset password as below

sql>alter user DBSNMP account unlock identified by oracle;

Do test connection and verify all information is correct.

9)Then configure ASM instances and you will be required to enter ASM password

10)We are done now. We can go to Targets ->Databases ->Tintin to view the database. It would look like below

Em12c:Silent Oracle Management agent Installation

This Post would describe how to install Oracle Management Agent 12c in silent mode. We need to download the agent software which can be done using emcli utility which is present on OMS host.Login to OMS host and execute emcli utility as mentioned below

[oracle@oradbdev04]~/Middleware/oms/bin% emcli login -username=sysman
Enter password 

Login successful

We need to now syncronize EMCLI

[oracle@oradbdev04]~/Middleware/oms/bin% emcli sync
Synchronized successfully

Following command gets list of agent software present on grid control

[oracle@oradbdev04]~/Middleware/oms/bin% emcli get_supported_platforms
Getting list of platforms ...
Check the logs at /home/oracle/Middleware/oms/bin/agent.log
About to access self-update code path to retrieve the platforms list..
Getting Platforms list  ...
-----------------------------------------------
Version = 12.1.0.1.0
 Platform = Linux x86-64
-----------------------------------------------
Platforms list displayed successfully.

Since this is a new Grid installation on Linux x86-64 we get only 1 agent software. If you need to install agent on different platform say Solaris, then you need to download software using self update option.
Note that you will have to pass your My oracle support credentials while configuring self-update.

Since our target host is also Linux x86-64 we can proceed. In below example we have specified that our 12.1.0.1.0 agent software should be placed in /tmp for Linux x86-64 platform

[oracle@oradbdev04]~/Middleware/oms/bin% ./emcli get_agentimage -destination=/tmp -platform="Linux x86-64" -version=12.1.0.1.0
Platform:Linux x86-64
Destination:/tmp
 === Partition Detail ===
Space free : 3 GB
Space required : 1 GB
Check the logs at /tmp/get_agentimage_2012-02-04_11-56-01-AM.log
Setting property ORACLE_HOME to:/home/oracle/Middleware/oms
calling pulloneoffs with arguments:/home/oracle/Middleware/oms/home/oracle/Middleware/oms/sysman/agent/12.1.0.1.0_AgentCore_226.zip12.1.0.1.0linux_x64
Check this logs for more information: /home/oracle/Middleware/oms/sysman/prov/agentpush/logs
[oracle@oradbdev04]~/Middleware/oms/bin% ls -l /tmp/12.1.0.1.0_AgentCore_226.zip
-rw-r----- 1 oracle dba 235679029 Feb  4 11:56 /tmp/12.1.0.1.0_AgentCore_226.zip

You should copy this file to the target host and unzip it.

To perform installation , we need to use agentDeploy.sh script. You can either use agent.rsp in unzipped directory or pass parameters at command line

AGENT_BASE_DIR ->Location where Agent software will be installed
OMS_HOST –>OMS host name
EM_UPLOAD_PORT –>Agent upload port. Script will try both http/https port.
AGENT_REGISTRATION_PASSWORD –>Password for secure registration with OMS

e.g
agentDeploy.sh AGENT_BASE_DIR=/home/oracle/agent12c OMS_HOST=oradbdev04 EM_UPLOAD_PORT=4900 AGENT_REGISTRATION_PASSWORD=oracle123

You will be prompted to run root.sh as below

-bash-3.2$ sudo /home/oracle/agent12c/core/12.1.0.1.0/root.sh
Password:
Finished product-specific root actions.
/etc exist

Creating /etc/oragchomelist file...
Finished product-specific root actions.

Verify following commands are working properly

cd $AGENT_HOME/core/12.1.0.1.0/bin
./emctl status agent
./emctl upload agent

$ ./emctl upload agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

Next step is to discover the targets. You can either manually add them or use automated discovery of new Oracle targets on machines which already have agent installed. Refer to EM12c:Automated discovery of Targets for automatic discovery and target configuration.

Get Upgrading: Upgrade to Enterprise Manager Cloud Control 12c

PreUpgradeConsole

This post describes the steps to upgrade an existing 11g Enterprise Manager Grid Control (11.1.0.1.0) to 12c (12.1.0.1.0). The Complete Upgrade process will be carried out under 3 different stages:

1. Patch existing 11g OMS to get the 12c upgrade console.
2. Deploy and Configure 12c agents.
3. Upgrade 11g OMS to 12c.

The approach used for this upgrade is 1-system upgrade approach. This approach enables you to upgrade to Enterprise Manager Cloud Control on the same host where your earlier release of Enterprise Manager is running. This approach also upgrades the Oracle Management Repository (Management Repository) in the existing database. Since the upgrade happens  on the same host, there is a reasonable downtime involved. This approach does not refer to upgrading of an Enterprise Manager system in an environment with one Oracle Management Service (OMS). It refers to upgrading of an Enterprise Manager system on the same host as the old one, and having only one Enterprise Manager system existing at any point. 

 Environment Details:

Operating System: OEL5.4 x86_64
Database Version: 11.2.0.1
Existing Grid Control/Agent: 11.1.0.1.0

The supported earlier releases for upgrade include:

Oracle Management Agent 10g Release 2 (10.2.x.x.x),
Oracle Management Agent 11g Release 1 (11.1.0.1.0),
Oracle Management Service 10g Release 5 (10.2.0.5.0) and
Oracle Management Service 11g Release 1(11.1.0.1.0).

Unlike earlier releases Oracle Management Service 12c communicates only with Oracle Management Agent 12c. Therefore, unlike the earlier releases, you must first upgrade the Management Agents in your system before upgrading your existing OMS.

1. Patch existing 11g OMS to get the 12c upgrade console

To start the upgrade process we need to get the 12c upgrade console, which we can get by installing patch for bug# 10044087 on your existing Enterprise Manager system.
To apply this patch OMS needs to be down which means that there is a downtime required for the existing GridControl environment.
Download the patch and apply for your platform click HERE for download link.

This patch application is a simple patch apply process with opatch. Following are the mandatory things before you start patch apply:

— It is mandatory to apply PSU1(10065631) or later on the existing OMS home

— There is a chance of hitting bug  9953045 and  bug 12360281, to escape these situations, set  SQLNET.ENCRYPTION_TYPES_SERVER= (3DES168)  in sqlnet.ora of your oracle database sqlnet file.

Follow the patch readme to apply this patch. Once this patch is applied and the post scripts(puc_dblink_pkgdef.sql, puc_dblink_pkgbody.sql and pre_upg_console.sql) are run and OMS is started, you can see the “Enterprise manager 12c Upgrade Console”  under “Deployments” tab of  Grid Control.

PreUpgradeConsole

2. Deploy and Configure 12c agents:

Download the following software, and stage them to an accessible location:

– Oracle Management Agent 12c
– All the required plug-ins

Click HERE to download.

Don’t unzip the binaries after downloading them. Verify the checksum after downloading the zip files.
To manage information about the location of the core Management Agent software and its associated plug-ins, follow these steps:

1. In Grid Control, click Deployments.
2. On the Deployments page, in the Upgrade section, click Enterprise Manager 12c Upgrade Console.
3. On the Upgrade Console page, in the Select Upgrade Type section, select one of the following:

– To upgrade your Enterprise Manager system with downtime (even if you manually install Management Agents), select 1-System.
– To upgrade your Enterprise Manager system with “near zero” downtime (even if you manually install Management Agents), select 2-System.
– To upgrade your Enterprise Manager system with downtime on a different host, select 1-System on a Different Host.

Select 1-system approach here.

Enterprise Manager Grid Control refreshes the page and displays a table with a list of tasks you need to perform for the upgrade approach you selected.

4. In the Preupgrade Steps section, from the table, click Manage Software.
5. On the Manage Software page, in the Agent Upgradability and Target Upgradability pie charts, click the hyperlinks in the respective legends and identify the Management Agents and targets that can be upgraded, and that cannot be upgraded due to unavailability of the Management Agent software or the plug-in software.

6. In the Provide Software Location section, enter the absolute path to the directory where the core Management Agent software and the plug-in software are present for the required platforms.

For example, /u01/app/plugin

And then click Validate to register that location with the Enterprise Manager system. Once validate is complete you will see the below pie charts adn these should be in all green. If they are not in green, it means that you have missed some plugin(you will find the list of all required plugins at the botton of same page), download and stage the missing plugin and validate again.

 

Validate

Validate

 

Once the software have been verified we can continue to deploy and configure the software binaries of Oracle Management Agent 12c.

1. In Grid Control, click Deployments.
2. On the Deployments page, in the Upgrade section, click Enterprise Manager 12c Upgrade Console.
3. On the Upgrade Console page, in the Select Upgrade Type section, select  1-system approach.

In the “Preupgrade Steps” section, from the table, click Deploy and Configure Agents.

 

 

4. On the Deploy and Configure Agents page, for Operation Name, enter a unique name for the deployment operation you are about to perform. The custom name you enter can be any intuitive name, and need not necessarily be in the same format as the default name.
For example, Deploy_Agents_Phase1_2010_12_27.
6. In the Select Operation Type section, select Deploy Agent and Plug-In Software
and Configure Agent and Plug-In Software.
7. In the Search Agents section, search and add the existing, old Management Agents
for which you want to deploy the software. In the table that lists the Management Agents, enter an installation base directory
and an instance home directory for each of the Management Agents.

In the Agent Credentials section, retain the default selection, that is, Use Oracle Home Preferred Credentials,if you have already set these, so that the preferred credentials stored in the Management Repository can be used for this job.In the OMS Host and Port section, validate the name and the secure port of the host where you plan to install Oracle Management Service 12c. To change the values, click Edit. After filling up the port details click next, do not select any pre and post upgrade script.
Provide the root credentials and click submit. Agent will start a job with the name that you have provided, click at the job to see the status.

 Generate Health Report of Deployed Oracle Management Agents.  Follow the steps by clicking the “Generate help report” , once it is completed, you will see following screen:

Verify and sign off the health check report. After this you have to Switch-Over to 12c agents. Under “Agent Upgrade Steps” click switch agents:

Search for all the agents and provide the oracle preferred credentials. Click submit. 

 Once this job completes, the old agents will be stopped and new agents from 12g home will be started.

 3. Upgrade 11g OMS to 12c

Download and unzip the 12c software from the link mentioned earlier in the post. It is mandatory to meet all the required prerequisites mentioned at ORACLE DOCUMENTATION otherwise the installation will fail at some later stage. I will list one import prerequisite step here:

Copy EMkey to repository: To do so, check the status of emkey as:
[oracle@cloud1 bin]$ emctl status emkey
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password : xxxxx
The EMKey is configured properly.

Copy the emkey as:

     $ emctl config emkey -copy_to_repos

After meeting all the prerequisites, continue with the OMS upgrade:

Invoke runInstaller from 12c unzip location:

 $ ./runInstaller

Choose your option to receive updates via MOS and click next. I opted to remain uninformed from MOS updates.

On the Software Updates screen, select one of the following sources from where the software updates can be installed while the Enterprise Manager system gets upgraded. If you do not want to apply them now, then select Skip. I opted to skip these updates for the installation. Click next.

 

On the Prerequisite Checks screen, check the status of the prerequisite checks run by the installation wizard, and verify whether your environment meets all the minimum requirements for a successful upgrade. It may show “Failed” for some of the packages, make sure that those were installed or higher versions of those packages are already in place. Once confirmed all settings, click next.

Under Install Type, select to upgrade an existing instance option and choose 1-system upgrade approach. It will automatically find the existing middleware home. Enter a valid middleware home location where it will install the required components for 12c. Provide the location of Middleware home . Click next.

Provide sys and sysman password and click next.

You can select the plug-ins that you want to install, the already installed plugins will be greyed-out.
Click next after selecting the plugins.

 

If you are upgrading from Enterprise Manager 11g Grid Control Release 1 (11.1.0.1.0), then on the Extend WebLogic Server Domain screen, validate the AdminServer host name and its port, and the WebLogic user name. Enter the WebLogic user account password for extending the existing Oracle WebLogic Server Domain to the upgraded release.

 

Installation status.. once the installation finishes, it will ask you to run allroot.sh as root user.
Run the script as root user and click finish.

Launch the Cloud Console as you will see like:

Login using sysman and now you are ready to control clouds 🙂