With this in mind it was decided to call the conference DevOpsDays. Videos of the talks will be online in the archives section soon, so I’ve decided to write down my thoughts about what went well and not so well – I am a fan of retrospectives.

What Went Well

The conference was in Ghent, a city in Belgium, and as expected had quite a heavy European contingent. It was not not exclusively so though, as Reductive Labs and Canonical both sent representatives over from the USA, and one of the speakers is an Aussie who just happened to be back packing around Europe at the time. One of my first great learnings though was that there are about 10 of us in the greater London area who are doing this kind of thing (2 of them ex-TWers) so we’ve started meeting regularly and exchanging ideas.

As there was such a large concentration of us around London, we’ve decided to try and subvert one of the London Geek Nights as a DevOps night to try and spread the word to more Devs. I’ll probably make more noise about it here when we’ve got dates confirmed.

The next great thing I loved was the format. It was split up into talks in the morning, with Open Spaces in the afternoon, over 2 days. This format actually worked very well as many of the open space sessions were inspired by the talks. Another interesting mix was that there was an even split between talks on tools and talks on practices.

I also discovered some great tools! Here’s a brief list of ones I liked the most:

• The Marionette Collective – using a message bus to administer 100′s of servers using pub/sub topics. You’ve got to watch the video to appreciate the full awesomeness.
• Cucumber-Nagios – writing tests to monitor your system using Cucumber, and then formatting the results as a Nagios plugin so you can easily wire it in to your monitoring system.
• Flapjack – another example of interesting use of a message bus. This time to parallelize the monitoring of 100′s of servers. I’m not sure I totally agree with the approach, but what really blew me away was they way they do graphing and reporting. It’s very impressive.
• collectd – collectd gathers statistics about the system it is running on and stores this information in RRD databases. One of the things I really like about this though is that there are plugins for it like Jcollectd which requires no code changes. Simply plug it into your classpath and it registers a JMX MBean that pumps data from your JVM to collectd (I think). I’ve not tried it in production yet though…

Overall the conference was very well received, and I’m looking forward to next years instalment. There’s a lot of talk about doing a North American version…

Not So Well

As we all know, the best bits about Away Days/Conferences are the talks in the hall between sessions. The problem with the program was that we barely had enough time for bio breaks between sessions, and everyone was quite burned out at the end of day two (not just because of the Belgian Beer). I think it’s a pretty easy thing to fix though and considering that it was the first conference of its type there’s bound to be some fine tuning required.

Recently though I’ve been speaking to more and more people who are trying to take an existing AMI and customise it for their own use. They do this by booting the AMI they want to base theirs on, doing the customization, then bundling up that volume. Generally they do pretty well, but there are three common themes that crop up that often cause pain: transient runtime configuration being bundled up, the time (and to a lesser extent effort) it takes to bundle the new image in the first place and making further changes to the image down the line.

Thankfully there is a simple single solution to these three problems – bundle from an image, not a running volume, and keep that image (or a set of images) along with some nice helper scripts on an EBS volume. That’s the theory, but as always there’s something in the real world that stops it being easy. By default, only the owner of an image can download and unpack an image directly from S3 and the images are encrypted with the owners EC2 private key. For this process to work, you’ll need to at least bootstrap yourself initially by going through the well known and well documented process of bundling a running system. After that though it’s easy. Really. Promise…

Let’s have a closer look at the problems we’re trying to solve first though before I go into how we fix them.

Transient Runtime Pollution

The most frustrating of these is the udev system flagging the source machines MAC address for eth0 and so making their custom image unusable because the network interface does not come up. There are still distributions out there which try to be “helpful” by remembering which physical device, such as network card, maps to which logical device name, such as eth0. This is not in itself a bad thing. This is one feature I was crying for 10 years ago when I started using Linux on bigger iron. I would dread adding another network card to a server because I would normally end up having to re-label the external interfaces. The thing is though that you’re now creating an image that could be running anywhere and you don’t have physical or even console access to it.

Other examples of things that break are helper scripts. Because we’re now on an operating system image that is meant to be able to run anywhere, there are certain things you want to run only once the very first time the system boots. Once they’ve run the these scripts either create a lock file, clear their own executable bits or even delete themselves. If you’re trying to re-bundle and image you’ve already booted, you need to make sure you back out these changes.

Doing your customization in an image that has never actually been booted helps you keep all these things pristine.

Time and Effort of Creation

This one is actually quite straight forward. When you’re bundling a running volume, what happens under the hood is:

1. A new sparse file for the image is created
2. A new filesystem is created on the new image file
3. This new filesystem is mounted somewhere
4. The contents of your running volume is copied into the new image file
5. The new filesystem is then unmounted
6. The image file is compressed and encrypted
7. The compressed and encrypted file is then split into chunks
8. Your manifest is created

Some of these steps very I/O intensive. When you’re working with an image though, steps 1 to 5 don’t happen (well, steps 3 and 5 are needed for you to make changes) so you’ll be doing almost 50% less IO. This means that bundling a new image will take about half time. If you work with your image on an EBS volume it’ll be even faster as they have better performance characteristics than the standard instance stores.

Bundling and uploading images are not simple commands though. You need to specify things like your AWS access key and provide your EC2 encryption key. There’s options for which kernels and ramdisks to use. There’s lots of typing which means lots of room for human error. The way to get around this is to have small shell scripts with all these options in them. Now they are simple commands…

Maintenance

Once you’ve got your new AMI looking the way you want and doing the things you need, chances are that a few weeks after you’ve started using it you find that there’s a security fix or package update you’d like to apply. Often this ends up with people starting the whole process from scratch again. Boot up a new instance of the AMI you want to update, update it, type in all those commands and remember the options you used to bundle the volume and upload the new one. If you kept your scripts and image on an EBS you could simply attach it to a running instance and make the fixes there using the same scripts you used last time. Hows that for repeatability?

“So, just how do I work with an image then?” I hear you ask. Here’s a basic outline to get you started.

1. Set up your environment

These steps assume that you have the EC2 AMI and API tools installed locally, and that you’re running the commands on an EC2 instance. If you don’t have them, please look at EC2 AMI Tools and EC2 API Tools.

You also need some environment variables configured to make life easier:

EC2_PRIVATE_KEY=/path/to/your/EC2/private/key
EC2_CERT=/path/to/your/EC2/cert

2. Create your EBS Volume

The hardest thing will be working out how big you need to make it. Absolute worst case will be 20gb per image, but in reality 10gb should be plenty. Remember though that an EBS can only be mounted in the availability zone it is created in, so this command creates one in the same zone you are in.

ec2-create-volume -s 10 -z `curl http://169.254.169.254/2008-09-01/meta-data/placement/availability-zone`

3. Prepare the Volume

First, attach the volume to a running EC2 instance. Make sure it’s at least the same type (i386 or x86_64) as the image you’re working on.

ec2-attach-volume vol-<your vol id> -i `curl http://169.254.169.254/2008-09-01/meta-data/instance-id` -d /dev/sdp

An EBS volume is a raw bit bucket. You need to partition it (if you’re in to that kind of thing) and create the filesystem on it. Partitions don’t really make sense here though, so just create a nice shiny filesystem on it once it’s mounted on the instance.

mke2fs -j /dev/sdp

In this instance I’m making an EXT3 filesystem, but you can use any filesystem that’s supported by the host machine. Please make sure though that the block device you specify (in this example /dev/sdp) matches what you told EC2 to mount your EBS volume on.

mkdir /ebs

mount /dev/sdp /ebs

This mounts your new filesystem on a directory called /ebs

mkdir /ebs/mnt

mkdir /ebs/download

mkdir /ebs/upload

mkdir /ebs/.ec2

This creates some handy directories to help you along with the process. This is what they do:

• mnt: Will be used as the mount point to access your image
• download: This is where you’ll to download your initial bundle to
• upload: When you bundle an image, put it here ready to be uploaded
• .ec2: This will contain your AWS access keys and your EC2 PEM files as follows:
  • s3.secret: S3 Secret Key
  • s3.access: S3 Access Key
  • ec2-pk.pem: EC2 Private Key
  • ec2-cert.pem: EC2 Certificate
  • id: EC2 user ID (Note: AWS account number, NOT Access Key ID)

ec2-download-bundle -b your-bucket -a `cat /ebs/.ec2/s3.access` -s `cat /ebs/.ec2/s3.secret` -k /ebs/.ec2/ec2-pk.pem -d /ebs/download -p your-image-name

This command pulls down the bundle you want to customise from S3. As I said before though, this will only work if you have sufficient rights on S3 to download the image and the EC2 private key that bundled it up and encrypted it in the first place.

ec2-unbundle -k /ebs/.ec2/ec2-pk.pem -s /ebs/download -d /ebs -m /ebs/download/your-image-name.manifest.xml

This command uncompresses and decrypts the image file from the downloaded bundle. It takes a while…

4. Customise

Now you’re ready to work. All you need is two shell scripts to go in your /ebs directory. work.sh mounts up your image (if it’s not mounted already) and chroot’s you in and you’re now up and running – customise to your hearts content. When you’re done, make sure you’ve logged out of all your work.sh scripts (yes you can run more than one) and then run bundle-and-upload.sh.

When you’re done, just shut down your host machine. When you want to work on it again later, just boot up a new AMI, attach your volume, mount it up and you’re at step 4 already.

Have fun…

The only criticism I have is that they don’t put enough emphasis on one of my key reasons for why it’s suddenly taken off. Cloud computing is not a new idea – it’s an extension of the Utility Computing that John McCarthy talked about in 1961. Although they only make a passing remark in section 3, I think one of the most important reasons it’s taken off is that the services Amazon provide were the first that were not a “solution looking for a problem”. Earlier offerings by the likes of Sun, HP and Intel all created a solution that they tried to sell to clients. The problem was that there were remarkably few problems that their solutions solved. Amazon simply exposed services that they were using internally already. That’s not to say the other reasons they give are not valid, I totally agree with them. I think they just missed a good point.

One of the topics I only glanced over is covered cover quite well in section 6 – Cloud Computing Economics. They provide some interesting example cost calculations. Although the numbers are obviously US centric, they do provide a nice way for a company to approach making the old “build vs buy” comparison.

In summary, I highly recommend this paper for anyone who wants to get the head around what this Cloud stuff is all about and what they need to do to prepare for it.

The Buildix image is now available for public use as ami-e4ca2f8d.

Here’s a couple of tips to keep in mind when rolling your own:

Remeber it’s Xen

I had problems getting my first batch of uploads working on the network. They would boot without a problem, but the networking would not initialize (as I could see from the console output). Things got a lot easier when I started manipulating the image locally using my own Xen installation. By booting it up on my local Xen server I could see where the problems were and fix them with a lot less pain and suffering.

The other benefit you get from testing your image in Xen is that you can bundle the image directly (ec2-bundle-image) instead of having to scrape a running machine (ec2-bundle-vol).

There can’t be only one

As the name implies, EC2 is all about having a big cloud of images. Chances are there will be a lot of instances of each image running out there. This means that the image itself needs to pick up its identity on boot up. Amazon provide a nice RESTful API that can provide each instance with information about itself. The documentation for the meta-data that’s available is available at http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/AESDG-chapter-instancedata.html. The stuff I needed the most was the public SSH key to allow to log in as root, and the hostname of the machine. Here’s what I added to /etc/rc.local to allow me to do this:

if [ ! -d /root/.ssh ] ; then
    mkdir -p /root/.ssh
    chmod 700 /root/.ssh
fi
# Fetch public key using HTTP
curl http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key > /tmp/my-key
if [ $? -eq 0 ] ; then
    cat /tmp/my-key >> /root/.ssh/authorized_keys
    chmod 600 /root/.ssh/authorized_keys
    rm /tmp/my-key
fi

hostname `curl http://169.254.169.254/latest/meta-data/hostname`

As always, there were also people who were interested in hearing about how it can be used for non Java projects. I had a good chat to one person who is interested in using it on a mixed Common LISP and Erlang project he’s working on. I’m looking forward to hearing how it goes for him. Due to lack of experience on my part I could unfortunately not help him much with the darcs problems he’s having though. Some people have all the fun…

It was also quite useful to speak to people about the problems they’re currently facing when trying to use CruiseControl. A common theme is people trying to manage large numbers of builds, or trying to build products across large numbers of different platforms. These are problems the dedicated ThoughtWorks development team are currently working on, so it’s great to get the validation that we’re putting effort into the things people care about now.

The whole point of Buildix is to help any Java based Agile Development Project get up and running as quickly as possible buy providing them with pre-configured and integrated version control system, continuous integration framework, wiki and issue tracking system. We chose our favourite products in each of these areas – Subversion, CruiseControl and Trac. Another common difficulty faced by our development teams, especially in the early stages of a project, is network access. Sometimes all we get from our clients when we arrive on site is a switch to allow us to get our laptops to talk to each other – no DNS, file shares, anything. Buildix can also help in situations like this, as it also runs Samba, and will run as a DNS and DHCP server if given the correct kernel boot parameters.

So, six months after we started, and after a few internal releases, we decided to give something back to the community that helps us do our job, and make Buildix available to everyone. Enjoy!

The first problem I bumped into was stability. Once I had configured and booted the zones I wanted, I discovered that ssh-keygen would segfault and dump core when run from within a zone. Normal ssh and scp commands would also occasionally segfault as well. After some very light scratching around, I decided to patch the systems to see if that made the problem go away. It did not – so I replaced it with the Sun Freeware OpenSSH package. This is when I found the next problem – patching.

For understandable reasons, Sun have restricted access to patches for Solaris 10. The days of just pulling down the latest recommended patch cluster to sort out your machines have gone. Sun now recommend you use Update Manager, a Java GUI app that registers your machine with Sun, and lists what patches you can download. All sounds reasonable in theory, but the first few times I tried it, it kept on blowing out with a com.sun.cns.authentication.CMDExecutionException. Turns out it’s broken on machines with Zones, and you need to manually download and apply a patch to fix it. Another thing to remember when patching is to make sure that all zones you have configured have been properly initialized, and that you’ve been through the system identifications at first boot.

And then we have the niggles category. There is no lsof package for amd64 Solaris 10 yet, but you can script most of what you need using pfiles and fuser. DTrace only works in the global zone. While the global zone has all the rights it needs to trace what’s going on, if you’re running more than two or three zones trying to find the specific process you’re trying to debug can be a pain.

With all this pain, is it worth it for development and testing? Most definitely! Zones allow you to have all the production-like environments you need for testing, or even just for developers to spike ideas in. Clients are happy because they don’t need to buy so much hardware. Testers are happy because if they need a new environment they can have it in a matter of hours instead of days or weeks. Sys Admins are happy because they don’t have to keep finding rack space for more machines.