ec2

Linux: ssh equivalence and SELinux

This is quick post summarizing issues encountered while setting up ssh equivalence on EC2 instance. I was setting up  RHEL7 EC2 instances and followed below procedure to setup ssh equivalence

  • Generate rsa key-pair using ssh-keygen -t rsa on both hosts
  • Copy the public keys to the remote server in authorized_keys file
  • Modify file permission to 600

But when I tried to perform ssh to remote host , it failed with following error.

Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

I verified directory (.ssh) and file permissions were correct. Then, I checked for SELinux context using ls – Z option.

-bash-4.2$ ls -lZ *
-rw-r--r--. postgres postgres unconfined_u:object_r:postgresql_db_t:s0 authorized_keys
-rw-------. postgres postgres unconfined_u:object_r:postgresql_db_t:s0 id_rsa
-rw-r--r--. postgres postgres unconfined_u:object_r:postgresql_db_t:s0 id_rsa.pub
-rw-r--r--. postgres postgres unconfined_u:object_r:postgresql_db_t:s0 known_hosts

As per above output, these files are running with postgresql_db_t type context. I used getenforce to verify that SELinux was in enforcing mode on this host. It can also be verified by viewing contents of /etc/selinux/config .

# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of three two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted

I decided to use restorecon command to restores SELinux security context for files and directories to their default values .

-bash-4.2$ restorecon -Rv /var/lib/pgsql/.ssh/
restorecon reset /var/lib/pgsql/.ssh context unconfined_u:object_r:postgresql_db_t:s0->unconfined_u:object_r:ssh_home_t:s0
restorecon reset /var/lib/pgsql/.ssh/id_rsa context unconfined_u:object_r:postgresql_db_t:s0->unconfined_u:object_r:ssh_home_t:s0
restorecon reset /var/lib/pgsql/.ssh/id_rsa.pub context unconfined_u:object_r:postgresql_db_t:s0->unconfined_u:object_r:ssh_home_t:s0
restorecon reset /var/lib/pgsql/.ssh/authorized_keys context unconfined_u:object_r:postgresql_db_t:s0->unconfined_u:object_r:ssh_home_t:s0
restorecon reset /var/lib/pgsql/.ssh/known_hosts context unconfined_u:object_r:postgresql_db_t:s0->unconfined_u:object_r:ssh_home_t:s0

-bash-4.2$ ls -lZ *
-rw-r--r--. postgres postgres unconfined_u:object_r:ssh_home_t:s0 authorized_keys
-rw-------. postgres postgres unconfined_u:object_r:ssh_home_t:s0 id_rsa
-rw-r--r--. postgres postgres unconfined_u:object_r:ssh_home_t:s0 id_rsa.pub
-rw-r--r--. postgres postgres unconfined_u:object_r:ssh_home_t:s0 known_hosts

As you can see, restorecon restored permission by changing type from postgresql_db_t to ssh_home_t. I performed ssh again and it worked !

Reference – https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/sect-security-enhanced_linux-working_with_selinux-selinux_contexts_labeling_files

 

Recovering from Lost AWS EC2 Key

This article is for AWS users using EC2 instances.
Summary: Please backup your AWS EC2 Key file and you will never have to go through below pain 🙂

AWS allows you to create a key-pair which can be downloaded on your machine and it is used for logging into your EC2 instance.
I had been using the same for accessing the EC2 instance for this blog. But last week my corporate laptop was upgraded and I took backup of all files except this key file.

I didnt realize this until saturday when I wanted to login to my EC2 instance. When I couldn’t find the file, I thought of downloading it again from AWS console.
But it was no where to be found. I searched and found out that we can’t download this file again.

Only solution to this problem is to create new EC2 instance with new key pair and then move site . I have documented high level tasks which can be used to restore. These steps are performed on AWS micro instance and its assumed that all your files are lying on EBS volume. My earlier article on AWS setup can be used for configuring EC2 instance.

1. Create new ec2 instance with new key-pair and make multiple backup copies of this key file and store at multiple location. Next install apache,php,mysql or any other component required for your site. Its better to create EC2 instance in same region as your old instance like us-east-1a.
2. Assign new elastic ip address to this EC2 instance.
3. If your site is working then you can login to wordpress panel and take mysql backup which can be restored to the new ec2 instance. I use WP-DBManager plugin for mysql backups.
4. Next take snapshot of your existing ec2 volume (hosting blog document directory) . This has to be done via EC2 dashboard ->Elastic Block Store ->Snapshots .This has to be in same region as new EC2 instance as we will be mounting the volume. On completion of snapshot, create a new volume out of this snapshot.
5. We need to attach this volume to new EC2 instance.  Go to EC2 dashboard ->Elastic Block Store ->Volumes. Choose the new volume and actions->attach volume and choose new EC2 instance name. Give volume name as /dev/xvdh
6. Within seconds, you should see this volume in your EC2 instances. e.g

cat /proc/partitions 
major minor  #blocks  name

 202        1    8388608 xvda1
 202      112    8388608 xvdh

7. Create a new directory and mount this volume
#mkdir /newdir
#mount /dev/xvdh /newdir

8. You can copy your blog files from /newdir/var/www/html (document root). Also any config files can be restored from old machine.
9. Import the mysql database from mysql dump
10. Once this is done, you can test whether your site is working fine.
e.g

UPDATE wp_options SET option_value = replace(option_value, 'http://oldsite', 'http://ec2-instance-name') WHERE option_name = 'home' OR option_name = 'siteurl';
UPDATE wp_posts SET guid = replace(guid, 'http://oldsite','http://ec2-instance-name');
UPDATE wp_posts SET post_content = replace(post_content, 'http://oldsite', 'http://ec2-instance-name');

This way you can verify if site is working fine. Once done restore back the old name by modifying above sql or reimporting the mysql db.

11. Final step is to switch site to new EC2. I am using cloudflare for DNS management which allows instant propagation of DNS change.So I assigned new elastic IP to this EC2 instance and changed my DNS record to point to this new EC2 instance.
Other option could be that you can disassociate your old elastic IP from old EC2 and attach to the new EC2 instance.