Automating AWS builds with cloud-init

How to ensure that when you launch an instance in AWS, it complies with you standard build and will do things like picking up the correct hostname, will connect with puppet and register with satellite, etc

This example is based on RedHat Enterprise Linux but can be adapted to other linux distributions.

Step 1 - set up the AWS CLI

I've used the Linux based AWS CLI in this procedure. Thic can be set up as follows:

1. Download zipped bundle (from link on http://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-bundle-other-os )
2. unzip awscli-bundle.zip
3. Install: awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
4. Now it needs to be configured:
  $ aws configure
  AWS Access Key ID [None]: as provided when your user was set up
  AWS Secret Access Key [None]: as provided when your user was set up
  Default region name [None]: eu-west-1
  Default output format [None]: ENTER

NOTE: uses python, must be version 2.6 or above

There is also a power shell based CLI should you be that way inclined.

Step 2 - creating an Amazon Machine Image (AMI) from your linux standard build

By creating an AMI based on your linux standard build, you can ensure that your linux instances comply with your company standards. One way of doing this is as follows:

1. Build a VMware VM using your standard build procedure
2. Export a VMDK file from the VM , in vcenter, select your VM > click file > Export > Export OVF Template > select OVF . You should end up with a VMDK file.
3. Copy the VMDK to a S3 bucket:  aws s3 cp my-vm-disk1.vmdk s3://justsomestuff/my-vm-disk1.vmdk
     #should see message
     upload: .\my-vm-disk1.vmdk to s3://justsomestuff/my-vm-disk1.vmdk
4. Import it: 
     aws ec2 import-image --cli-input-json "{  \"Description\": \"JustSomeStuff Base Image\", \"DiskContainers\": [ { \"Description\": \"First CLI task\", \"UserBucket\": { \"S3Bucket\": \"justsomestuff\", \"S3Key\" : \"my-vm-disk1.vmdk\" } } ]}"
5. This takes a while. You can monitor progress as follows:
     aws ec2 describe-import-image-tasks --cli-input-json "{ \"ImportTaskIds\": [\"import-ami-fgo894kw\"], \"NextToken\": \"abc\", \"MaxResults\": 10 } "
   where the ImportTasksIds is the id of the task from the import-image command

Once the import has finished, you should see an AMI called “JustSomeStuff Base Image” listed in you browse your AMIs via AWS Management Console

Step 3 - Customise the base AMI for cloud

Now that you have an AMI based on your standard build, you can use this to launch instances. However, if you launched an instance as is from the AMI, it would have the same hostname, SSH keys, etc, as the VM it was created from. To change this you can use a feature called cloud-init. First off you need to customise your AMI slightly for cloud use

1. Launch an instance from your base AMI (assume the id is ami-1234d1e2)
     aws ec2 run-instances --image-id ami-1234d1e2 --count 1 --instance-type t2.micro --key-name JustSomeStuff --security-group-ids sg-1a2b3c4d  --subnet-id subnet-1a2b3c4d
2. Log into the instance and install the cloud-init rpm package (this is included in the epel66-x86_64 stream)
3. Once the cloud-init package is installed cd /etc/cloud/templates/
4. Edit hosts.redhat.tmpl adding entries for the puppet and satellite servers, e.g.

#*
This file /etc/cloud/templates/hosts.redhat.tmpl is only utilized
if enabled in cloud-config.  Specifically, in order to enable it
you need to add the following to config:
manage_etc_hosts: True
*#
# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.redhat.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
# The following lines are desirable for IPv4 capable hosts
127.0.0.1 ${fqdn} ${hostname}
127.0.0.1 localhost.localdomain localhost
127.0.0.1 localhost4.localdomain4 localhost4
10.40.160.100   uksatellite.justsomestuff.co.uk uksatellite
10.40.160.100   puppet.justsomestuff.co.uk      puppet
# The following lines are desirable for IPv6 capable hosts
::1 ${fqdn} ${hostname}
::1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

5. Create an update AMI from this updated instance
      aws ec2 create-image --instance-id i-a13b9929 --name "JustSomeStuff Base Image with cloud-init" --description "JustSomeStuff standard RedHat 6.6 build with cloud-init and customised hosts template"
      

The quick witted amongst you may be asking, why not do this when you build the VM (step 2). Well the answer is, that would be sensible. However, we had already created our standard AMI so I didn't want to go back and create it again (I know, how lazy…)

Step 4 - launch instances

 1. Create a cloud-init file. Here's the one I used
 
 #cloud-config
 # == Hostname management (via /etc/hostname) ==
 hostname: jss01
 fqdn: jss01.justsomestuff.co.uk
 manage_etc_hosts: True
 # == Regenerate ssh host keys, register with satellite and make sure pupet up to date ==
 runcmd:
   - rm -f /etc/ssh/ssh_host_rsa_key
   - rm -f /etc/ssh/ssh_host_dsa_key
   - ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa
   - ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa
   - rm -f /root/.bash_history
   - cd /var/tmp && wget http://10.44.173.76/post/satellite_register.sh && sed -i -e  's/\. ${POST_RUN}/\# \. ${POST_RUN}/' satellite_register.sh && sh satellite_register.sh
   - puppetd --test
 
 the commands listed under the runcmd section. These will recreate the SSH host keys, upload a satellite register script to /var/tmp and run it, then make sure puppet 
 downloads the required files.
 The puppet part requires you to have puppet installed in your base image and for you to create a manifest entry on the puppet master first.
  
  2. Launch the instance using the cloud-config file:
        aws ec2 run-instances --image-id ami-1234d1e2 --user-data file:///var/tmp/cloud-init --count 1 --instance-type t2.micro --key-name JustSomeStuff --security-group-ids sg-1a2b3c4d  --subnet-id subnet-1a2b3c4d
 

Once the image has launched, it will have the a hostname of jss01, you'll be able to install packages from satellite and all the puppet controlled files will be in place.

You can tag the instance with a name as follows:

 aws ec2 create-tags --resources i-4da871c5 --tags Key=Name,Value=jss01

You can also use the cloud-config file in the AWS Management Console by cutting and pasting it into the userdata box when you launch an instance.

cloud-init is also supported with Openstack.

Recent Changes