Build EBS-Backed CentOS EC2 AMI from Scratch

Published: April 09, 2014 Last Updated: October 01, 2017 Author: Saad Ali

WARNING! Following this article, improvise if necessary. Your environment may be different than mine. I am not responsible if you screw up!

In this tutorial I will demonstrate how to create a custom CentOS 6 EC2 AMI from scratch. Most of this tutorial contains work already done by Jeffrey M. Hunter. The original article can be found here. I improvised on a few steps where I was having issues following the original article.

Motivation

AWS marketplace already has CentOS 6 AMIs. First problem my colleagues said was that root partition of the instance is 8G only and cannot be resized (we wanted to install cPanel/WHM on the instance). We took care of that by increasing the partition size while launching a new instance and resizing the partition online:

# resize2fs /dev/xvde

Second problem was that the instances with AWS Marketplace codes cannot be attached to any other instance should the need arise. I found this link where an AWS Marketplace engineer has commented that "they are aware of the issue". That post is dated 4th October, 2012 and if it is to be believed that the post is indeed written by an Amazon marketplace engineer then Amazon has still done nothing to solve this issue at the time of publishing of this post.

So in the light of the above circumstances, we thought to create our own CentOS 6 AMI. After all, Amazon does provide us all the needed tools. We created a build environment on top of CentOS 6.

Prerequisites

Given below are prerequisites to create an EC2 image:

AWS account with EC2, S3 and EBS services.
AWS account number.
AWS Access Key ID and Secret Access Key.
EC2 Private Key File and EC2 Certificate File (you'll have to generate one).
An EC2 micro instance to create an EBS backed AMI.

Setup Build Environment

Add the following environment variables to root user's .bashrc file:

    export EC2_HOME=/opt/ec2/tools
    export EC2_PRIVATE_KEY=/opt/ec2/certificates/ec2-pk.pem
    export EC2_CERT=/opt/ec2/certificates/ec2-cert.pem
    export EC2_URL=https://ec2.amazonaws.com
    export AWS_ACCOUNT_NUMBER=<000000000000>
    export AWS_ACCESS_KEY_ID=<your_access_key_id>
    export AWS_SECRET_ACCESS_KEY=<your_secret_access_key>
    export AWS_AMI_BUCKET=AMI/CentOS6
    export PATH=$PATH:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:$EC2_HOME/bin
    export JAVA_HOME=/usr

Either logout and log back in for the changes to take effect or make your changes effective immediately:

# source /root/.bashrc

Create and download EC2 Private key and Certificate files and copy them to /opt/ec2/certificates/:

# mkdir -p /opt/ec2/certificates
# cp pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem /opt/ec2/certificates/ec2-pk.pem
# cp cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem /opt/ec2/certificates/ec2-cert.pem

Download and setup EC2 API tools:

# mkdir -p /opt/ec2/tools
# curl -o /tmp/ec2-api-tools.zip http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
# unzip /tmp/ec2-api-tools.zip -d /tmp
# cp -r /tmp/ec2-api-tools-*/* /opt/ec2/tools

Download and setup EC2 AMI tools:

# curl -o /tmp/ec2-ami-tools.zip http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip
# unzip /tmp/ec2-ami-tools.zip -d /tmp
# cp -rf /tmp/ec2-ami-tools-*/* /opt/ec2/tools

Install required packages as follows:

# yum -y install e2fsprogs ruby java-1.6.0-openjdk unzip MAKEDEV

Finally, setup a yum configuration file /opt/ec2/yum/yum.conf with the following contents (you may change the release version in the file):

[base]
name=CentOS - Base
mirrorlist=http://mirrorlist.centos.org/?release=6.5&arch=x86_64&repo=os
#baseurl=http://mirror.centos.org/centos/6.5/os/x86_64/
gpgcheck=1
enabled=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

[updates]
name=CentOS - Updates
mirrorlist=http://mirrorlist.centos.org/?release=6.5&arch=x86_64&repo=updates
#baseurl=http://mirror.centos.org/centos/6.5/updates/x86_64/
gpgcheck=1
enabled=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

[extras]
name=CentOS - Extras
mirrorlist=http://mirrorlist.centos.org/?release=6.5&arch=x86_64&repo=extras
#baseurl=http://mirror.centos.org/centos/6.5/extras/x86_64/
gpgcheck=1
enabled=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-6

Prepare AMI

start by creating an empty 1G ext4 file system image and mount it on loopback:

# dd if=/dev/zero of=/opt/ec2/images/CentOS-6-Base-x86_64.img bs=1M count=1024
# mkfs.ext4 -F -j -L 'ROOTFS' /opt/ec2/images/CentOS-6-Base-x86_64.img
# mkdir -p /mnt/ec2-image
# mount -o loop /opt/ec2/images/CentOS-6-Base-x86_64.img /mnt/ec2-image/

Create directories in the new root file system to hold system files and devices:

# mkdir -p /mnt/ec2-image/{dev,etc,proc,sys}
# mkdir -p /mnt/ec2-image/var/{cache,log,lock,lib/rpm}

Populate /dev on new root file system with a minimal set of devices. Ignore any MAKEDEV: mkdir: File exists warnings and then do bind mounts in the new root file system:

    # /sbin/MAKEDEV -d /mnt/ec2-image/dev -x console
    # /sbin/MAKEDEV -d /mnt/ec2-image/dev -x null
    # /sbin/MAKEDEV -d /mnt/ec2-image/dev -x zero
    # /sbin/MAKEDEV -d /mnt/ec2-image/dev -x urandom
    # mount -o bind /dev /mnt/ec2-image/dev
    # mount -o bind /dev/pts /mnt/ec2-image/dev/pts
    # mount -o bind /dev/shm /mnt/ec2-image/dev/shm
    # mount -o bind /proc /mnt/ec2-image/proc
    # mount -o bind /sys /mnt/ec2-image/sys

Install CentOS Base operating system and a few other necessary packages in the root file system:

    # yum -c /opt/ec2/yum/yum.conf --installroot=/mnt/ec2-image -y groupinstall Base
    # yum -c /opt/ec2/yum/yum.conf --installroot=/mnt/ec2-image -y install openssh-server openssh-clients dhclient grub e2fsprogs yum-plugin-fastestmirror.noarch selinux-policy selinux-policy-targeted

We only needed one root file system partition so we defined file system table in /mnt/ec2-image/etc/fstab as follows:

LABEL=ROOTFS    /               ext4            defaults        1    1
none            /dev/pts        devpts          gid=5,mode=620  0    0
none            /dev/shm        tmpfs           defaults        0    0
none            /proc           proc            defaults        0    0
none            /sys            sysfs           defaults        0    0

Create grub configuration file /mnt/ec2-image/boot/grub/grub.conf in the root file system:

default=0
timeout=0
title CentOS (x86_64)
root (hd0)
kernel /boot/vmlinuz ro root=LABEL=ROOTFS
initrd /boot/initramfs

Modify the grub file as follows to add kernel version to the init images:

# kern=`ls /mnt/ec2-image/boot/vmlin*|awk -F/ '{print $NF}'`
# ird=`ls /mnt/ec2-image/boot/initramfs*.img|awk -F/ '{print $NF}'`
# sed -ie "s/vmlinuz/$kern/" /mnt/ec2-image/boot/grub/grub.conf
# sed -ie "s/initramfs/$ird/" /mnt/ec2-image/boot/grub/grub.conf

Create a symlink to menu.lst:

# ln -s /boot/grub/grub.conf /mnt/ec2-image/boot/grub/menu.lst

Modify grub.conf file so that kernel version is appended:

# kern=`ls /mnt/ec2-image/boot/vmlin*|awk -F/ '{print $NF}'`
# ird=`ls /mnt/ec2-image/boot/initramfs*.img|awk -F/ '{print $NF}'`
# sed -ie "s/vmlinuz/$kern/" /mnt/ec2-image/boot/grub/grub.conf
# sed -ie "s/initramfs/$ird/" /mnt/ec2-image/boot/grub/grub.conf

Add .bashc and .bash_profile to root's directory:

/mnt/ec2-image/root/.bashrc

    # .bashrc

    # User specific aliases and functions

    alias rm='rm -i'
    alias cp='cp -i'
    alias mv='mv -i'

    # Source global definitions
    if [ -f /etc/bashrc ]; then
            . /etc/bashrc
    fi

/mnt/ec2-image/root/.bash_profile

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

Configure network options for the image by adding the following 2 files:

/mnt/ec2-image/etc/sysconfig/network

    NETWORKING=yes
    HOSTNAME=localhost.localdomain

/mnt/ec2-image/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE="eth0"
NM_CONTROLLED="yes"
ONBOOT=yes
TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no

CentOS comes with SELinux set to enforcing by default; however, in some cases doesn't get labelled correctly depending on the instance being created. It is best to assume that for the first start of the instance that it is not properly labelled. Run the following to ensure labelling is executed on the first start of the instance:

# touch /mnt/ec2-image/.autorelabel

If you want to disable SELinux in the new image, just edit /mnt/ec2-image/etc/sysconfig/selinux as follows:

SELINUX=disabled

Modify /mnt/ec2-image/etc/rc.local to add SSH key to the system for pem based logins:

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

# set a random pass on first boot
if [ -f /root/firstrun ]; then
    dd if=/dev/urandom count=50|md5sum|passwd --stdin root
    passwd -l root
    rm /root/firstrun
fi

if [ ! -d /root/.ssh ]; then
    mkdir -m 0700 -p /root/.ssh
    restorecon /root/.ssh
fi
# Get the root ssh key setup
ReTry=0
while [ ! -f /root/.ssh/authorized_keys ] && [ $ReTry -lt 5 ]; do
    sleep 2
    curl -f http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key > /root/.ssh/authorized_keys
    ReTry=$[Retry+1]
done
chmod 600 /root/.ssh/authorized_keys && restorecon /root/.ssh/authorized_keys

Modify /mnt/ec2-image/etc/ssh/sshd_config for SSH configuration as follows:

UseDNS no
PermitRootLogin without-password

Configure the image to run network and SSH services on instance startup:

# /usr/sbin/chroot /mnt/ec2-image /sbin/chkconfig --level 2345 network on
# /usr/sbin/chroot /mnt/ec2-image /sbin/chkconfig --level 2345 sshd on

Clean up and unmount the image:

# yum -c /opt/ec2/yum/yum.conf --installroot=/mnt/ec2-image -y clean packages
# rm -rf /mnt/ec2-image/root/.bash_history
# rm -rf /mnt/ec2-image/var/cache/yum
# rm -rf /mnt/ec2-image/var/lib/yum
# umount /mnt/ec2-image/dev/shm
# umount /mnt/ec2-image/dev/pts
# umount /mnt/ec2-image/dev
# umount /mnt/ec2-image/sys
# umount /mnt/ec2-image/proc
# umount /mnt/ec2-image

Build AMI

When we go to register the AMI with Amazon EC2 later in this guide, we must set the default kernel as one which supports the GRUB boot loader. To enable user-provided kernels, Amazon has published Amazon Kernel Images (AKIs) that use a system called PV-GRUB. PV-GRUB is basically a boot manager for XEN virtual machines. Given below is a quote from the original article as to how it works:

PV-GRUB is a paravirtual "mini-OS" that runs a version of GNU GRUB, the standard Linux boot loader. PV-GRUB selects the kernel to boot by reading /boot/grub/menu.lst from your image which we configured earlier in this guide. It will load the kernel specified by your image (the CentOS kernel) and then shut down the "mini-OS", so that it no longer consumes any resources. One of the advantages of this solution is that PV-GRUB understands standard grub.conf or menu.lst commands, which allows it to work with most existing Linux distributions.

Find the latest available AKI:

    # ec2-describe-images --owner amazon --region us-east-1 | grep "amazon\/pv-grub-hd0" | awk '{ print $1, $2, $3, $5, $7 }'

If you encounter an error as follows:

Client.InvalidSecurity: Request has expired

do update system date.

If you have followed the tutorial uptill now you can safely use an AKI with hd0 in the name for a unpartitioned image. If you image has a partition table you must use an AKI with hd00 in the name. At the time of this writing, the latest available AKI is aki-919dcaf8. Use the following command to generate an AMI bundle:

# ec2-bundle-image --cert $EC2_CERT --privatekey $EC2_PRIVATE_KEY --image /opt/ec2/images/CentOS-6-Base-x86_64.img --prefix CentOS-6-Base-x86_64 --user $AWS_ACCOUNT_NUMBER --destination /opt/ec2/images --arch x86_64 --kernel aki-919dcaf8

Upload the AMI bundle to S3 bucket:

# ec2-upload-bundle --manifest /opt/ec2/images/CentOS-6-Base-x86_64.manifest.xml --bucket $AWS_AMI_BUCKET --access-key $AWS_ACCESS_KEY_ID --secret-key $AWS_SECRET_ACCESS_KEY

At this point you can register the new AMI (the uploaded AMI) bundle using:

# ec2-register $AWS_AMI_BUCKET/CentOS-6-Base-x86_64.manifest.xml --name "CentOS (x86_64)" --description "CentOS (x86_64) Base AMI" --architecture x86_64 --kernel aki-919dcaf8

and this will start and instance store-backed instance.

There are a few additional steps required to have EBS backing for an AMI.

Register an EBS-Backed AMI

For an EBS-backed AMI, we are gonna use a t1.micro instance. Start a t1.micro instance (with any Linux distribution) using AWS web console. Download private key file on the instance in /opt directory and setup a temporary environment as follows:

# export EC2_PRIVATE_KEY=/opt/ec2-pk.pem
# export AWS_AMI_BUCKET=AMI/CentOS6
# export AWS_ACCESS_KEY_ID=<your_access_key_id>
# export AWS_SECRET_ACCESS_KEY=<your_secret_access_key>

Download the AMI bundle:

# ec2-download-bundle -b $AWS_AMI_BUCKET -m CentOS-6-Base-x86_64.manifest.xml -a $AWS_ACCESS_KEY_ID -s $AWS_SECRET_ACCESS_KEY --privatekey $EC2_PRIVATE_KEY -d /opt

Unbundle AMI:

# ec2-unbundle -m CentOS-6-Base-x86_64.manifest.xml --privatekey $EC2_PRIVATE_KEY

This will extract the original image that you created. Using AWS web console, create a volume (this will act as the hard drive volume for the AMI) of the required size (we created a 150G volume), attach it to the t1.micro instance and write the image file on the colume using dd:

# dd if=/opt/CentOS-6-Base-x86_64 of=/dev/xvdf bs=1M

You can since the image created was 1G in size, you might wanna expand to make more space available to the OS in the image using resize2fs and scan using e2fsck:

# resize2fs /dev/xvdf
# e2fsck -f /dev/xvdf

Detach the volume from the instance and using AWS web console:

Create a snapshot of the volume.
Create an AMI from the snapshot.

Now you can launch the new EC2 instance using the custom AMI backed by EBS.

Instance SWAP

You can create an 8G SWAP file on the instance as follows:

# dd if=/dev/zero of=/swapfile bs=1M count=8192
# mkswap /swapfile
# chmod 0600 /swapfile
# swapon /swapfile

Add swap to /etc/fstab as follows:

/swapfile       swap            swap            defaults        0    0

Tagged as: Amazon AWS AMI CentOS EBS EC2 VPC Linux S3