Unix

Linux: ssh equivalence and SELinux

This is quick post summarizing issues encountered while setting up ssh equivalence on EC2 instance. I was setting up  RHEL7 EC2 instances and followed below procedure to setup ssh equivalence

  • Generate rsa key-pair using ssh-keygen -t rsa on both hosts
  • Copy the public keys to the remote server in authorized_keys file
  • Modify file permission to 600

But when I tried to perform ssh to remote host , it failed with following error.

Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

I verified directory (.ssh) and file permissions were correct. Then, I checked for SELinux context using ls – Z option.

-bash-4.2$ ls -lZ *
-rw-r--r--. postgres postgres unconfined_u:object_r:postgresql_db_t:s0 authorized_keys
-rw-------. postgres postgres unconfined_u:object_r:postgresql_db_t:s0 id_rsa
-rw-r--r--. postgres postgres unconfined_u:object_r:postgresql_db_t:s0 id_rsa.pub
-rw-r--r--. postgres postgres unconfined_u:object_r:postgresql_db_t:s0 known_hosts

As per above output, these files are running with postgresql_db_t type context. I used getenforce to verify that SELinux was in enforcing mode on this host. It can also be verified by viewing contents of /etc/selinux/config .

# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of three two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted

I decided to use restorecon command to restores SELinux security context for files and directories to their default values .

-bash-4.2$ restorecon -Rv /var/lib/pgsql/.ssh/
restorecon reset /var/lib/pgsql/.ssh context unconfined_u:object_r:postgresql_db_t:s0->unconfined_u:object_r:ssh_home_t:s0
restorecon reset /var/lib/pgsql/.ssh/id_rsa context unconfined_u:object_r:postgresql_db_t:s0->unconfined_u:object_r:ssh_home_t:s0
restorecon reset /var/lib/pgsql/.ssh/id_rsa.pub context unconfined_u:object_r:postgresql_db_t:s0->unconfined_u:object_r:ssh_home_t:s0
restorecon reset /var/lib/pgsql/.ssh/authorized_keys context unconfined_u:object_r:postgresql_db_t:s0->unconfined_u:object_r:ssh_home_t:s0
restorecon reset /var/lib/pgsql/.ssh/known_hosts context unconfined_u:object_r:postgresql_db_t:s0->unconfined_u:object_r:ssh_home_t:s0

-bash-4.2$ ls -lZ *
-rw-r--r--. postgres postgres unconfined_u:object_r:ssh_home_t:s0 authorized_keys
-rw-------. postgres postgres unconfined_u:object_r:ssh_home_t:s0 id_rsa
-rw-r--r--. postgres postgres unconfined_u:object_r:ssh_home_t:s0 id_rsa.pub
-rw-r--r--. postgres postgres unconfined_u:object_r:ssh_home_t:s0 known_hosts

As you can see, restorecon restored permission by changing type from postgresql_db_t to ssh_home_t. I performed ssh again and it worked !

Reference – https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/sect-security-enhanced_linux-working_with_selinux-selinux_contexts_labeling_files

 

How To Configure Exadata Database Machine in Enterprise Manager Cloud Control 13c (OEM13c)

I have followed the steps in Oracle Documentation link: https://docs.oracle.com/cd/E63000_01/EMXIG/ch2_deployment.htm#EMXIG215 to configure Exadata Database Machine in OEM13c. If you want to configure your Exadata in OEM13c you have to follow the above mentioned link.
In this post I will share the mandatory steps for configuration, and some of the issues which I faced while configuring the Exadata on OEM13c.
NOTE: OEM13c agents only needs to be deployed on compute nodes on your Exadata Machine.

Step 1: Deploy Exadata Plug-In in OEM13c.

Step 2: For an EM agent to communicate with ILOM SP, there must be a user created on ILOM SP, on all the Compute Nodes.
Create A Database Server ILOM SP (Service Processor) User.
Login to Compute Node ILOM with “root” user”
# cd /SP/users
# create oemuser
Creating user…
Enter new password: ********
Enter new password again: ********

Created /SP/users/oemuser

Change to the new user’s directory and set the role:

# cd oemuser
/SP/users/oemuser

set role=’cro’
Set ‘role’ to ‘cro’

Now test the ILOM user ID created:

For Exadata X5-2:
# ipmitool -I lanplus -H <ComputeNodeILOMHostname> -U oemuser -P xxxxxx -L USER sel list last 10
It should display some results.

Now run the above steps on all Compute Node ILOMs.

STEP 3: Push the OEM agent to Compute nodes.
From OEM13c console, select Setup from top right corner, and then Add Target, and the Add Target Manually. Put the Compute Node’s hostname, and then select your OS version. Fill-in the rest of the details on the screen and click Deploy.  It will Deploy the agent on the Compute Nodes you have mentioned.

 

Step 4: Run discovery Precheck Script:
To ensure that discovery of Exadata Machine complete without any issues, you need to run exadataDiscoveryPreCheck.pl. This script is available under OEM13c OMS server Exadata plug-in location i.e:
<OMS_agent installation directory>/plugins/oracle.sysman.xa.discovery.plugin_12.1.0.3.0/discover/dbmPreReqCheck/exadataDiscoveryPreCheck.pl, verify the path as per your configuration and run the script. You can also download the script from MOS Note: 1473912.1.

NOTE: For Infiniband user you have to use “nm2user” and its default password is changeme.

This script showed following errors to me:
Verifying setup files consistency... 
------------------------------------ 
Verifying cell nodes... 
Cell node <CellNode Name> is missing in one of the setup files. 
Cell node <CellNode Name>.domain is missing in one of the setup files. 
Cell node <CellNode Name>.domain is missing in one of the setup files. 
Cell node <CellNode Name> is missing in one of the setup files. 
Cell node <CellNode Name> is missing in one of the setup files. 
Cell node <CellNode Name>.domain is missing in one of the setup files. 
Verifying infiniband nodes... 
Infiniband node <IBNode Name>.domain is missing in one of the setup files. 
Infiniband node <IBNode Name> is missing in one of the setup files. 
Infiniband node <IBNode Name>.domain is missing in one of the setup files. 
Infiniband node <IBNode Name> is missing in one of the setup files. 
Infiniband node null is missing in one of the setup files. 
Verifying KVM nodes... 
KVM node null is missing in one of the setup files. 
Verifying PDU nodes... 
PDU node <PDUNode Name> is missing in one of the setup files. 
PDU node <PDUNode Name> is missing in one of the setup files. 
PDU node <PDUNode Name>.domain is missing in one of the setup files. 
PDU node <PDUNode Name>.domain is missing in one of the setup files. 
Setup files are not consistent ===> Not ok 
* Please make sure that node information in both parameter and schematic files 
is consistent. 
======================================================= 
* Please make sure ciphers are correctly set in all cell and compute nodes. 
Verifying SSH cipher definition for <CellNode Name> cell node... 
None of the expected ciphers were found in sshd_config file ===> Not ok 
* Please make sure ciphers are correctly set in sshd_config file. 
== =========================================================

 So there were two issues:
1. Parameter file and Schematic file were not in sync with each other.
2. Missing valid cipher in cellnodes’ sshd_config file.
For parameter files issue, we need to check two files under /opt/oracle.SupportTools/onecommand, em.params and databasemachine.xml, and have to make sure that entries are same
in these files. In my case all the names under em.params were with fqdn and under databasemachine.xml these were without fqdn. I modified em.params to remove the fqdn from
all names.
For cipher issue, as the compute nodes did not error out for valid ciphers, I have copied one cipher entry from Compute Node to all the Cell Nodes and restarted the sshd service.

After making these two changes I ran exadataDiscoveryPreCheck.pl script again and it came out clean.

STEP 5: Discovering an Exadata Database Machine

1. From the Enterprise Manager home page, select the Setup menu (upper right corner), Add Target, and then Add Targets Manually.

2. On the Add Targets Manually page, click Add Targets Using Guided Process. From Add Using Guided Process window, select Oracle Exadata Database Machine from the list and click Add.

3. On the Oracle Exadata Database Machine Discovery page, select one of the following tasks:
13c target type
12c target type
I opted for 13c target type.

4. On the Discovery Inputs page, enter the following information
For the Discovery Agents section:
Agent URL: The Agent deployed on compute node. Click the search icon to select from available URLs.
For the Schematic Files section:
Once you have specified the Agent URL, a new row (hostname and schematic file information) is automatically added. The default schematic file, databasemachine.xml, describes the hardware components of the Exadata Database Machine.
Click Set Credential to set the credentials for the host.
Check/modify the schematic file location.
Select the schematic file name from drop-down menu.

5. On the InfiniBand Discovery page, enter the following information:
IB Switch Host Name: The InfiniBand switch host name. The IB Switch host name is usually pre-populated.
InfiniBand Switch ILOM host credential: The user name (usually ilom-admin or ilom-operator) and password for the InfiniBand switch ILOM host.

Rest of the steps are self explanatory and can be filled easily.

On Credentials page, after filling root password, you will get two options under SNMP credentials:
— Credential Type SNMPV1
— Credential Type SNMPV3
I opted for SNMPV3 and it requires EXACLI username/password. So you have to create ExaCli user as described at
http://docs.oracle.com/cd/E50790_01/doc/doc.121/e50471.pdf on Page 384.
Create the Exacli user and provide the information asked under SNMPV3.

Click Submit and it will take some time to discover the Exadata DB Machine.

After this, you can see “Exadata” under Targets tab on OEM13c home page.

 

portmap: unrecognized service on RHEL6

Quick note for people using NFS for shared storage on RAC database. Till RHEL5 we had to ensure nfs,nfslock and portmap service has to be running.
These services are required otherwise you will get following errors while mounting database

ORA-00210: cannot open the specified control file
ORA-00202: control file: '/u01/oradata/orcl/control01.ctl'
ORA-27086: unable to lock file - already in use

Mostly this could be auto-enabled on boot by using chkconfig command. While working on similar issue today, I found out that this service is not present in RHEL 6

# service portmap status
portmap: unrecognized service

The portmap service was used to map RPC program numbers to IP address port number combinations in earlier versions of Red Hat Enterprise Linux.
As per RHEL6 docs, portmap service has been replaced by rpcbind in Red Hat Enterprise Linux 6 to enable IPv6 support.
So following command will work

# service rpcbind status
rpcbind (pid  1587) is running...

You can read about NFS and associated processes from RHEL6 docs

Installing Adobe Flash Player using Yum on Linux

I have been using firefox on a linux desktop since last week. Lot of sites on internet are using flash player on it , so everytime I hit the site, it reported missing plugin. When I tried to install the plugin through firefox, it errored out and asked me to install manually. Finally today I decided to install it using manual approach. To do so I went to http://www.adobe.com/go/getflashplayer to download the required binaries.

Surprisingly today I noticed a yum based install for it. I thought of  going with yum as it takes care of resolving dependency. On selecting yum based rpm, it downloaded adobe-release-x86_64-1.0-1.noarch.rpm. This rpm needs to be installed

 rpm -ivh adobe-release-x86_64-1.0-1.noarch.rpm
warning: adobe-release-x86_64-1.0-1.noarch.rpm: Header V3 DSA signature: NOKEY, key ID f6777c67
Preparing...                ########################################### [100%]
   1:adobe-release-x86_64   ########################################### [100%]

Next step is to use yum to install the flash-player. Before that I did search to check the plugin using yum search flash-plugin

flash-plugin.i386 : Adobe Flash Player 10.3 i386
flash-plugin.i386 : Adobe Flash Player 10.3 i386
flash-plugin.i386 : Adobe Flash Player 10.3 i386
flash-plugin.i386 : Adobe Flash Player 10.3 i386
flash-plugin.i386 : Adobe Flash Player 10.3 i386
flash-plugin.i386 : Adobe Flash Player 10.3 i386
flash-plugin.i386 : Adobe Flash Player 10.3 i386
flash-plugin.x86_64 : Adobe Flash Player 11.2
flash-plugin.i386 : Adobe Flash Player 10.3 i386
flash-plugin.i386 : Adobe Flash Player 10.2 i386
flash-plugin.i386 : Adobe Flash Player 10.3 i386

Finally installed it using yum install flash-plugin and restarted firefox

Parsing package install arguments
Resolving Dependencies
--> Running transaction check
---> Package flash-plugin.x86_64 0:11.2.202.233-release set to be updated
---> Package flash-plugin.i386 0:10.3.183.19-1.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
 flash-plugin            x86_64     11.2.202.233-release  adobe-linux-x86_64  6.9 M
 flash-plugin            i386       10.3.183.19-1.el5  updates           4.9 M

Transaction Summary
=============================================================================
Install      2 Package(s)
Update       0 Package(s)
Remove       0 Package(s)         

Total download size: 12 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): flash-plugin-10.3.183.19-1.el5.i386.rpm                                                                | 4.9 MB     00:07
(2/2): flash-plugin-11.2.202.233-release.x86_64.rpm                                                           | 6.9 MB     00:48
warning: rpmts_HdrFromFdno: Header V3 DSA signature: NOKEY, key ID f6777c67
Importing GPG key 0xF6777C67 "Adobe Systems Incorporated (Linux RPM Signing Key) " from /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux
Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing: flash-plugin                 ######################### [1/2]
  Installing: flash-plugin                 ######################### [2/2] 

Installed: flash-plugin.x86_64 0:11.2.202.233-release flash-plugin.i386 0:10.3.183.19-1.el5
Complete!

Cluster SSH tool Utility

Many times you come across scenario’s when you wish to open multiple ssh windows and execute same commands.e.g You wish to see alert log for multi-node RAC simultaneously or edit sysctl.conf files for multiple machines.

Cluster SSH utility helps solve this problem as it opens multiple SSH sessions and allows simultaneous control.

In case you are using MAC OS X, you can download from Google Code site . Utility is called csshX.

For Linux you can download from Sourceforge site and utility is called Cluster SSH.

I am using csshX to show demo. Suppose I have 3 hosts host1,host2,host3 I need to use following commands on terminal app

$csshX host1 host2 host3

or

$csshX host[1-3]

As you can see we can specify range using square brackets.  csshX will create an SSH session to each remote host in separate Terminal.app windows. A master window will also be created. All keyboard input in the master will be sent to all the slave windows. Below screenshot displays how windows will look like

I can enter commands in Master (Red Color) window and it will execute same commands in all 3 windows.If you want to execute commands in particular window then go directly  to that window.  In case we need to open 3 sessions for host1 then we execute

$csshx host1+3

Use awk/sed in vi

Thought of sharing some useful info which can help you to do your work faster in vi.You can use awk/sed scripts in vi using following command in vi

:%!scriptname

Here the scriptname file should have execute privilges for user. I used this to create a useful script which I was doing by typing multiple substitution command in vi.

e.g Your file contains list of table
$cat t.lst
BANK005
BJSTM
BJS_ORG
CHAINED_ROWS
CORR_BAM
CORR_CAM
CORR_EIT
CORR_GAC
CORR_GAM
CORR_ITC
CORR_LDT
CORR_LHT
Create script (quotes)  with following command and give execute permission to user.

sed -e “s/^/’/g” -e “s/$/’,/” $1|awk ‘{printf (“%s”,$0)}’|sed -e “s/^/(/g” -e “s/,$/)/g”

open t.lst in vi and type :%!quotes
('BANK005','BJSTM','BJS_ORG','CHAINED_ROWS','CORR_BAM','CORR_CAM','CORR_EIT','CORR_GAC','CORR_GAM','CORR_ITC','CORR_LDT','CORR_LHT')

Similarly if you wish to remove blank lines, have a file blank like

awk ‘!NF==0 {print $0}’ $1
Blank lines can also be directly removed from vi using :g/^$/d
Isn’t it cool.. 🙂