How To Configure Exadata Database Machine in Enterprise Manager Cloud Control 13c (OEM13c)

I have followed the steps in Oracle Documentation link: https://docs.oracle.com/cd/E63000_01/EMXIG/ch2_deployment.htm#EMXIG215 to configure Exadata Database Machine in OEM13c. If you want to configure your Exadata in OEM13c you have to follow the above mentioned link.
In this post I will share the mandatory steps for configuration, and some of the issues which I faced while configuring the Exadata on OEM13c.
NOTE: OEM13c agents only needs to be deployed on compute nodes on your Exadata Machine.

Step 1: Deploy Exadata Plug-In in OEM13c.

Step 2: For an EM agent to communicate with ILOM SP, there must be a user created on ILOM SP, on all the Compute Nodes.
Create A Database Server ILOM SP (Service Processor) User.
Login to Compute Node ILOM with “root” user”
# cd /SP/users
# create oemuser
Creating user…
Enter new password: ********
Enter new password again: ********

Created /SP/users/oemuser

Change to the new user’s directory and set the role:

# cd oemuser
/SP/users/oemuser

set role=’cro’
Set ‘role’ to ‘cro’

Now test the ILOM user ID created:

For Exadata X5-2:
# ipmitool -I lanplus -H <ComputeNodeILOMHostname> -U oemuser -P xxxxxx -L USER sel list last 10
It should display some results.

Now run the above steps on all Compute Node ILOMs.

STEP 3: Push the OEM agent to Compute nodes.
From OEM13c console, select Setup from top right corner, and then Add Target, and the Add Target Manually. Put the Compute Node’s hostname, and then select your OS version. Fill-in the rest of the details on the screen and click Deploy.  It will Deploy the agent on the Compute Nodes you have mentioned.

 

Step 4: Run discovery Precheck Script:
To ensure that discovery of Exadata Machine complete without any issues, you need to run exadataDiscoveryPreCheck.pl. This script is available under OEM13c OMS server Exadata plug-in location i.e:
<OMS_agent installation directory>/plugins/oracle.sysman.xa.discovery.plugin_12.1.0.3.0/discover/dbmPreReqCheck/exadataDiscoveryPreCheck.pl, verify the path as per your configuration and run the script. You can also download the script from MOS Note: 1473912.1.

NOTE: For Infiniband user you have to use “nm2user” and its default password is changeme.

This script showed following errors to me:
Verifying setup files consistency... 
------------------------------------ 
Verifying cell nodes... 
Cell node <CellNode Name> is missing in one of the setup files. 
Cell node <CellNode Name>.domain is missing in one of the setup files. 
Cell node <CellNode Name>.domain is missing in one of the setup files. 
Cell node <CellNode Name> is missing in one of the setup files. 
Cell node <CellNode Name> is missing in one of the setup files. 
Cell node <CellNode Name>.domain is missing in one of the setup files. 
Verifying infiniband nodes... 
Infiniband node <IBNode Name>.domain is missing in one of the setup files. 
Infiniband node <IBNode Name> is missing in one of the setup files. 
Infiniband node <IBNode Name>.domain is missing in one of the setup files. 
Infiniband node <IBNode Name> is missing in one of the setup files. 
Infiniband node null is missing in one of the setup files. 
Verifying KVM nodes... 
KVM node null is missing in one of the setup files. 
Verifying PDU nodes... 
PDU node <PDUNode Name> is missing in one of the setup files. 
PDU node <PDUNode Name> is missing in one of the setup files. 
PDU node <PDUNode Name>.domain is missing in one of the setup files. 
PDU node <PDUNode Name>.domain is missing in one of the setup files. 
Setup files are not consistent ===> Not ok 
* Please make sure that node information in both parameter and schematic files 
is consistent. 
======================================================= 
* Please make sure ciphers are correctly set in all cell and compute nodes. 
Verifying SSH cipher definition for <CellNode Name> cell node... 
None of the expected ciphers were found in sshd_config file ===> Not ok 
* Please make sure ciphers are correctly set in sshd_config file. 
== =========================================================
 So there were two issues:
1. Parameter file and Schematic file were not in sync with each other.
2. Missing valid cipher in cellnodes’ sshd_config file.
For parameter files issue, we need to check two files under /opt/oracle.SupportTools/onecommand, em.params and databasemachine.xml, and have to make sure that entries are same
in these files. In my case all the names under em.params were with fqdn and under databasemachine.xml these were without fqdn. I modified em.params to remove the fqdn from
all names.
For cipher issue, as the compute nodes did not error out for valid ciphers, I have copied one cipher entry from Compute Node to all the Cell Nodes and restarted the sshd service.

After making these two changes I ran exadataDiscoveryPreCheck.pl script again and it came out clean.

STEP 5: Discovering an Exadata Database Machine

1. From the Enterprise Manager home page, select the Setup menu (upper right corner), Add Target, and then Add Targets Manually.

2. On the Add Targets Manually page, click Add Targets Using Guided Process. From Add Using Guided Process window, select Oracle Exadata Database Machine from the list and click Add.

3. On the Oracle Exadata Database Machine Discovery page, select one of the following tasks:
13c target type
12c target type
I opted for 13c target type.

4. On the Discovery Inputs page, enter the following information
For the Discovery Agents section:
Agent URL: The Agent deployed on compute node. Click the search icon to select from available URLs.
For the Schematic Files section:
Once you have specified the Agent URL, a new row (hostname and schematic file information) is automatically added. The default schematic file, databasemachine.xml, describes the hardware components of the Exadata Database Machine.
Click Set Credential to set the credentials for the host.
Check/modify the schematic file location.
Select the schematic file name from drop-down menu.

5. On the InfiniBand Discovery page, enter the following information:
IB Switch Host Name: The InfiniBand switch host name. The IB Switch host name is usually pre-populated.
InfiniBand Switch ILOM host credential: The user name (usually ilom-admin or ilom-operator) and password for the InfiniBand switch ILOM host.

Rest of the steps are self explanatory and can be filled easily.

On Credentials page, after filling root password, you will get two options under SNMP credentials:
— Credential Type SNMPV1
— Credential Type SNMPV3
I opted for SNMPV3 and it requires EXACLI username/password. So you have to create ExaCli user as described at
http://docs.oracle.com/cd/E50790_01/doc/doc.121/e50471.pdf on Page 384.
Create the Exacli user and provide the information asked under SNMPV3.

Click Submit and it will take some time to discover the Exadata DB Machine.

After this, you can see “Exadata” under Targets tab on OEM13c home page.

 

16 July, 2016 by 12c Exadata GRID Unix 0

Is Your Data Center Ready For Exadata Machines!!

Before opting to procure Exadata machines, the first thing you need to check is the readiness of your data center to hold these machines. Exadata’s dimensions are similar to other normal racks i.e a 42U rack. The complete details for X5-2 physical site requirements can be found HERE.

This is very important step as, for some data centers its may take months to complete the site requirements and you do not want to put your Exadata in a store due to issues like space unavailability etc. in data centers. There are three important things that you must consider before buying it:

  1. There should be enough space in data center to hold this machine.
  2. The power requirement should be considered carefully as the machine will be customized to client specific requirement i.e single phase or three phase. Oracle will ask you, before you place the order, about your power settings in data center. You have to tell them that you want a machine with single phase or three phase power settings. I think this cannot be changed after machine’s delivery, but not sure.
  3. The most important one is the network requirements. Setting up Exadata will require some heavy pre-calculation for your network. The IPs, network switches, 10G uplinks, patch panels, Fiber ready modules in core switches. Only the management access is through 1G copper wire, whole client access should be done through 10G fiber connectivity.

Once all these things are ready in your data center, you are ready to deploy Exadata.

 

17 May, 2016 by Exadata 1

Exadata X6-2 Launched

5th April 2016, Oracle announced arrival of Exadata X6-2. Overview of all its capabilities can be found HERE. Check DataSheet for list of hardware components and capacity of each X6-2 rack.

Sometimes, Oracle changes hardware specifications for its racks e.g X5-2 was released with 4TB HDD initially, but after few months the datasheet was modified and Oracle started delivering X5-2 with 8TB HDD in storage cells. So, it is advisable to check the datasheet before ordering it to make sure that what you will be getting eventually.

 

8 April, 2016 by Exadata 0

Planning Database Hardware Upgrade !!! Consider Oracle Exadata

Upgrading database hardware in an organization is always a cumbersome process. The most time consuming step is, planing for the upgrade, which mainly includes choosing right hardware for your Oracle databases. After deciding on the hardware type for your databases, rest will be taken care by technical teams involved. I will be discussing, how I reached to the conclusion of implementing Exadata in my organization.

I am dealing with multiple hardware vendors(HP, IBM, Dell etc.) and different type of bare metal servers(Rack mounted, blade machines etc.). Also to add on, I do have virtualization for some databases. Most of the servers are old, and needs to be replaced. The reason was, we started to face issues like server end of life, frequent hardware failures, performance degradation etc. To start with the hardware upgrade process, I checked multiple solutions from different vendors like, HP Converged Systems, Oracle SuperCluster, all blade environment(quarter/half/full height), all virtualized  environment(Hyper-v, VMware, Oracle VM), Oracle Exadata, EMC v-Block etc.

I had to decide on the best solution based on TCO(Total Cost of Ownership) and annual maintenance cost. A significant time is spent analyzing and comparing all these solutions. After completing four months extensive study on financial impact and overall hardware performance, Exadata comes out to be the best of all because of following main reasons:

  • It reduced the space required for database servers in DataCenter. I will be removing at least four racks in my DataCenter with Exadata implementation.
  • Eliminate external SAN storage need for databases which save huge amount of money.
  • SAN switches( Cisco MDS etc.) not required for server to SAN connectivity. This is usually a hidden component in DataCenters which consumes a lot of money and is often a bottleneck for database I/Os if not configured properly.
  • Its COD feature(Introduce from X5-2) saves a lot of money if the resource requirement is less.
  • Licensing cost of Oracle Databases will be reduced as new DBs can be added without any additional cost.
  • Its Storage Software which will process the business logic and return the processed results to compute nodes. It completely changes the traditional processing of queries and move the query processing to storage servers.

The above mentioned reasons are only a few, there are many more benefits and new features which I will be discussing in my future posts for exadata.

If you are deciding to revamp your data-center for hosting databases, it is good to consider Oracle Exadata.

7 April, 2016 by Exadata 0

PROCESSED Messages not clearing from Oracle Queue

I was contacted by Dev team to look into Development database where Oracle queue size kept  on increasing. As per them messages were not getting cleared.They were running count on Queue table and the count kept increasing.
To find out exact cause I included the msg_state in the query and found out that there were lot of unprocessed messages

SQL> select queue,msg_state,count(*) from A$$JMS_QUEUE_TABLE group by queue,msg_state;

QUEUE                   MSG_STATE      COUNT(*)
------------------------------ ---------------- ----------
AQ$_JMS_QUEUE_TABLE_E      EXPIRED             3
JMS_QUEUE               PROCESSED           343
JMS_QUEUE               READY             3

Since this was dev box,I initially tried purging the queue using dbms_aqadm.purge_queue_table

DECLARE
po_t dbms_aqadm.aq$_purge_options_t;
BEGIN
dbms_aqadm.purge_queue_table('SCOTT.JMS_QUEUE_TABLE',
purge_condition => null,
purge_options => po_t);
END;

This removed the EXPIRED and READY messages but PROCESSED messages count didn’t decrease.
Next  I checked queue definition using all_queues.I saw a value named retention specified to 3600.

SQL> SELECT owner, name, retention FROM all_queues WHERE name LIKE '%JMS%';

OWNER                   NAME                  RETENTION
-------            ------------------        --------------
SCOTT               JMS_QUEUE              3600
SCOTT               AQ$_JMS_QUEUE_TABLE_E      0

When a queue is defined in Oracle, you can define how long a message can remain visible once it has been dequeued (i.e. the retention period). Modifying the retention to 0 cleared the messages instantly

exec dbms_aqadm.alter_queue(QUEUE_NAME=>'SCOTT.JMS_QUEUE',RETENTION_TIME=>0)

Some interesting learning for me 🙂

1 February, 2016 by 10g 11g 0

12.1.0.2 PDB fails to come out of restricted mode

This one is a nasty bug 🙂 I was trying to setup Oracle PDB in a test environment for the first time and got stuck with ORA-01035 error

[oracle@oracle11g ~]$ sqlplus hr/hr@//oracle11g:1522/engg

SQL*Plus: Release 12.1.0.2.0 Production on Fri Jul 3 07:34:58 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

ERROR:
ORA-01035: ORACLE only available to users with RESTRICTED SESSION privilege

Checking the status of this pluggable database, I could see that DB was open but it was in restricted mode

SQL>  select con_id,logins,status from v$instance;

    CON_ID LOGINS     STATUS
---------- ---------- ------------
         0 RESTRICTED OPEN

SQL> select con_id,name,open_mode from v$pdbs;

    CON_ID NAME                           OPEN_MODE
---------- ------------------------------ ----------
         2 PDB$SEED                       READ ONLY
         3 ENGG                           READ WRITE

--Had got following error while opening/creating PDB

SQL> alter pluggable database engg open;

Warning: PDB altered with errors.

I tried creating PDB manually / DBCA multiple times but all PDB remained in restricted state. Alert log also didn’t report any error or explanation for this errors. Tried searching on internet but didn’t find any relevant hit. Finally MOS pointed out (Bug 19174942) that this could happen in 12.1.0.2 ,if a common user has default tablespace which is not present in PDB. I knew this had to be my issue as I had created a common user with default tablespace users but PDB’s didn’t had this tablespace. Fix was to create this tablespace in all the PDB’s and restart the PDB

SQL> alter session set container=engg;

Session altered.

SQL> create tablespace users datafile '/oracle/oradata1/orcl12c/pdbseed/user_01.dbf' size 100m ;

Tablespace created.

SQL> alter pluggable database engg close;

Pluggable database altered.

SQL> alter pluggable database engg open;

Pluggable database altered.

SQL>  select instance_name,status,logins from v$instance;

INSTANCE_NAME    STATUS       LOGINS
---------------- ------------ ----------
orcl12c          OPEN         ALLOWED

 

3 July, 2015 by 12c 2

Disclaimer!!

This blog reflect our own views and do not necessarily represent the views of our current or previous employers.

The contents of this blog are from our experience, you may use at your own risk, however you are strongly advised to cross reference with Product documentation and test before deploying to production environments.

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 188 other subscribers