A lot of this extra cost comes from a perceived requirement for the highest available capacity, availability and performance. Achieving all three characteristics is expensive, but if you’re willing to sacrifice one of them then costs start to fall considerably. Lowering requirements on two of the three drops it even more.

One of the teams I work with has a requirement primarily on capacity. Performance and availability are nice, but capacity is the key. We generate gigabytes worth of log files every day, but didn’t have one place to store it all for easy analysis. Just before I joined the team they’d purchased the cheapest “Enterprise” storage system the IT team at the time would allow – it ended up costing in the region of £12k for 12TB of raw storage. That’s £1000 per TB!

In addition to the price, the other problems were accessibility and management of the data and managing growth. This inspired a hunt for something that would provide a cheaper and more flexible solution.

Our requirements were:

• *nix based system. The current storage solution was based on Windows Storage Server, but all our systems and tools for this team are Linux based. Yes, Windows does technically provide things like an NFS server, but fighting with the file system permissions and overall performance are two things that impacted us.
• Cheap to expand. We need to have a clear path to grow the storage in the server easily by simply adding more disks.
• Large filesystems. There’s nothing more wasteful from a storage point of view than having lots of small filesystems. Besides the management overhead, there’s also many wasted blocks lying around un-used.
• Cheap to build. This inevitably means commodity hardware.
• Reasonable availability. We don’t need 99.999% uptime, but would be happy with somewhere in the region of 90%+
• Reasonable performance. Primary access to the data on this machine is via gigabit Ethernet. As long as it can keep up with the network card we’re happy…

Hardware Options

The first thing we looked at was the Backblaze concept. Using a custom designed case (that’s actually quite easy to purchase) they manage to cram 45 SATA drives into a 4U chassis. That’s pretty impressive. Using cheap SATA hot swap port multiplier/backplanes and controllers allows for quite a low cost, but not without problems though. In the configuration they publish there are a number of problems:

• A mix of PCIe and PCI-X SATA controllers means that you need a LOT of slots on your motherboard. Boards with all those slots are not cheap.
• Each of the 45 drives is presented to the operating system (Linux in this case, which is good). They then use software RAID for the volumes. Care needs to be taken when building a system like this though to ensure that you spread the drives for each array evenly across controllers so that a single controller or expander backplane failure does not take out an entire volume, which just adds to the management overhead.
• The port multiplier chipset they use is only supported on Linux. This was a problem as we had initially planned to use OpenSolaris for this device as ZFS would make the storage management and expansion a no-brainer. The Oracle take over of Sun and their subsequent clamp down on OpenSolaris derailed this plan. I did do some performance testing with FreeBSD and their implementation of ZFS but the performance was terrible.
• The cost is also not that cheap. Single orders of the custom case and power supply that go with it are actually pretty pricy.
• Performance. I did a lot of testing, mostly with iozone. What I found though was that when running through the port multiplier, even when using NCQ, performance was pretty bad.

The system I was using for my testing though had an onboard SAS controller and port expander. The SAS specification includes SATA support. Out of curiosity I decided to compare performance on that controller to what I got from the SATA chain. It was actually a lot better, which got me thinking about using SAS controllers with hardware RAID to simplify administration.

Initially I tried a pair of 8 port LSI controllers (LSI 8208ELP), but the cheaper end of the range have issues in their firmware and do not even boot with more than one controller present, so I replaced them with a single Adaptec 51245 controller. This made management and expansion a lot easier, as it has enough internal ports for the initial drives we planned on, and an external port that allows easy expansion.

Current architecture

The current system we have running in production is assembled from 100% off the shelf components. They are:

Cost (September 2010): £2130.17 (Excl VAT)

Total Raw Space: 20TB

Unit Cost: £106.50 per TB

Power Draw: 0.8A with sustained disk access, 1.16A peak on boot up.

The operating system we’ve selected is Ubuntu 10.04 LTS. We run the smaller disks we had spare in RAID1 for the OS. The data drives are in RAID6 on the WD’s.

Hardware Build

Here are some photos we took as we built the system:

Operating System Configuration

Using the hardware RAID controller allows us to simplify the Operating System configuration. Instead of having a block device for each physical disk, the controller presents a single block device for each array.

The filesystem we’re using is EXT4, created with the following options:

-m 1 -O dir_index,has_journal,extent,sparse_super

To try and speed things up we also set the noatime flag when mounting the filesystem.

The Adaptec Storage Manager application is used to monitor the health of the disks and will email us in the case of a drive failure.

Performance

With the current configuration, the performance is in no way spectacular, but it’s more than good enough for our needs. Write speeds are quite slow peaking at around 90MB/s, reads are good peaking at around 750MB/s.

Client Access and Usage

We’re currently running a mixed workload on the system and so far have had no complaints or problems. Every night we rsync all of our log files from our production machines, which is currently about 300GB per day.

Some of the people who are interested in this data access it via HTTP. We run an Apache server on the box (with compression enabled of course) to allow people easy access to browse and download what they need. Developers also access the logs read-only via NFS, and have a directory mounted via NFS called “playpen” that they use as a scratch pad for any of their research that requires more disk space than the 300GB or so they have available in their development machines.

Going Forward

In the 6 months or so we’ve had the machine we’ve almost used all the disk space. On a clear disk you can seek forever. We’ve placed an order for 2 more disk trays (NetStor NS330S-8026). These trays have built in SAS port expanders and allow us to daisy chain up to 7 trays off our external port. We’ve also ordered different hard drives. Instead of using the green WD drives we’ve decided to go for a bit more speed and get 7200RPM Hitachi Ultrastar 7K3000 SATA drives. This does add a lot more to the cost, but the performance pay off is worth it for us for expected higher write speeds. Once they’re up and running I’ll publish some comparative benchmarks.

Costs for the upgrade (Feb 2011): £9699.00

Total Raw Space: 64TB

Unit Cost: £151.50 per TB

Update (23 June 2011)

Right now there is some kind of major compatibility problem between the Adaptec card and the NetStor JBOD. I’ve been wrestling for over a week now with vendor support and still not getting anywhere.

It’s by far the longest article I’ve ever written, and I was amazed at how ideas that had been floating around in my head for a while crystalized through the processes of writing them down. I found I got so passionate talking to people about what was in it that I’ve decided to make a talk out of it, the first iteration of which will be in Chicago on Tuesday (see previous post).

HTTP Request Returned 401 Unauthorized: Failed to authenticate!

A quick sift through Google results didn’t get anything usable. A quick sniff of the packets going over the wire though showed that it was authenticating using a signed certificate. Normally when you sign HTTP requests like that you add some kind of timed expiry. Could the problem be clock related?

Sure enough, a quick check on the new client and the server showed that there was just over an hour time difference. Getting the time on the client and the server in sync got the client registered!

With this in mind it was decided to call the conference DevOpsDays. Videos of the talks will be online in the archives section soon, so I’ve decided to write down my thoughts about what went well and not so well – I am a fan of retrospectives.

What Went Well

The conference was in Ghent, a city in Belgium, and as expected had quite a heavy European contingent. It was not not exclusively so though, as Reductive Labs and Canonical both sent representatives over from the USA, and one of the speakers is an Aussie who just happened to be back packing around Europe at the time. One of my first great learnings though was that there are about 10 of us in the greater London area who are doing this kind of thing (2 of them ex-TWers) so we’ve started meeting regularly and exchanging ideas.

As there was such a large concentration of us around London, we’ve decided to try and subvert one of the London Geek Nights as a DevOps night to try and spread the word to more Devs. I’ll probably make more noise about it here when we’ve got dates confirmed.

The next great thing I loved was the format. It was split up into talks in the morning, with Open Spaces in the afternoon, over 2 days. This format actually worked very well as many of the open space sessions were inspired by the talks. Another interesting mix was that there was an even split between talks on tools and talks on practices.

I also discovered some great tools! Here’s a brief list of ones I liked the most:

• The Marionette Collective – using a message bus to administer 100′s of servers using pub/sub topics. You’ve got to watch the video to appreciate the full awesomeness.
• Cucumber-Nagios – writing tests to monitor your system using Cucumber, and then formatting the results as a Nagios plugin so you can easily wire it in to your monitoring system.
• Flapjack – another example of interesting use of a message bus. This time to parallelize the monitoring of 100′s of servers. I’m not sure I totally agree with the approach, but what really blew me away was they way they do graphing and reporting. It’s very impressive.
• collectd – collectd gathers statistics about the system it is running on and stores this information in RRD databases. One of the things I really like about this though is that there are plugins for it like Jcollectd which requires no code changes. Simply plug it into your classpath and it registers a JMX MBean that pumps data from your JVM to collectd (I think). I’ve not tried it in production yet though…

Overall the conference was very well received, and I’m looking forward to next years instalment. There’s a lot of talk about doing a North American version…

Not So Well

As we all know, the best bits about Away Days/Conferences are the talks in the hall between sessions. The problem with the program was that we barely had enough time for bio breaks between sessions, and everyone was quite burned out at the end of day two (not just because of the Belgian Beer). I think it’s a pretty easy thing to fix though and considering that it was the first conference of its type there’s bound to be some fine tuning required.

Recently though I’ve been speaking to more and more people who are trying to take an existing AMI and customise it for their own use. They do this by booting the AMI they want to base theirs on, doing the customization, then bundling up that volume. Generally they do pretty well, but there are three common themes that crop up that often cause pain: transient runtime configuration being bundled up, the time (and to a lesser extent effort) it takes to bundle the new image in the first place and making further changes to the image down the line.

Thankfully there is a simple single solution to these three problems – bundle from an image, not a running volume, and keep that image (or a set of images) along with some nice helper scripts on an EBS volume. That’s the theory, but as always there’s something in the real world that stops it being easy. By default, only the owner of an image can download and unpack an image directly from S3 and the images are encrypted with the owners EC2 private key. For this process to work, you’ll need to at least bootstrap yourself initially by going through the well known and well documented process of bundling a running system. After that though it’s easy. Really. Promise…

Let’s have a closer look at the problems we’re trying to solve first though before I go into how we fix them.

Transient Runtime Pollution

The most frustrating of these is the udev system flagging the source machines MAC address for eth0 and so making their custom image unusable because the network interface does not come up. There are still distributions out there which try to be “helpful” by remembering which physical device, such as network card, maps to which logical device name, such as eth0. This is not in itself a bad thing. This is one feature I was crying for 10 years ago when I started using Linux on bigger iron. I would dread adding another network card to a server because I would normally end up having to re-label the external interfaces. The thing is though that you’re now creating an image that could be running anywhere and you don’t have physical or even console access to it.

Other examples of things that break are helper scripts. Because we’re now on an operating system image that is meant to be able to run anywhere, there are certain things you want to run only once the very first time the system boots. Once they’ve run the these scripts either create a lock file, clear their own executable bits or even delete themselves. If you’re trying to re-bundle and image you’ve already booted, you need to make sure you back out these changes.

Doing your customization in an image that has never actually been booted helps you keep all these things pristine.

Time and Effort of Creation

This one is actually quite straight forward. When you’re bundling a running volume, what happens under the hood is:

1. A new sparse file for the image is created
2. A new filesystem is created on the new image file
3. This new filesystem is mounted somewhere
4. The contents of your running volume is copied into the new image file
5. The new filesystem is then unmounted
6. The image file is compressed and encrypted
7. The compressed and encrypted file is then split into chunks
8. Your manifest is created

Some of these steps very I/O intensive. When you’re working with an image though, steps 1 to 5 don’t happen (well, steps 3 and 5 are needed for you to make changes) so you’ll be doing almost 50% less IO. This means that bundling a new image will take about half time. If you work with your image on an EBS volume it’ll be even faster as they have better performance characteristics than the standard instance stores.

Bundling and uploading images are not simple commands though. You need to specify things like your AWS access key and provide your EC2 encryption key. There’s options for which kernels and ramdisks to use. There’s lots of typing which means lots of room for human error. The way to get around this is to have small shell scripts with all these options in them. Now they are simple commands…

Maintenance

Once you’ve got your new AMI looking the way you want and doing the things you need, chances are that a few weeks after you’ve started using it you find that there’s a security fix or package update you’d like to apply. Often this ends up with people starting the whole process from scratch again. Boot up a new instance of the AMI you want to update, update it, type in all those commands and remember the options you used to bundle the volume and upload the new one. If you kept your scripts and image on an EBS you could simply attach it to a running instance and make the fixes there using the same scripts you used last time. Hows that for repeatability?

“So, just how do I work with an image then?” I hear you ask. Here’s a basic outline to get you started.

1. Set up your environment

These steps assume that you have the EC2 AMI and API tools installed locally, and that you’re running the commands on an EC2 instance. If you don’t have them, please look at EC2 AMI Tools and EC2 API Tools.

You also need some environment variables configured to make life easier:

EC2_PRIVATE_KEY=/path/to/your/EC2/private/key
EC2_CERT=/path/to/your/EC2/cert

2. Create your EBS Volume

The hardest thing will be working out how big you need to make it. Absolute worst case will be 20gb per image, but in reality 10gb should be plenty. Remember though that an EBS can only be mounted in the availability zone it is created in, so this command creates one in the same zone you are in.

ec2-create-volume -s 10 -z `curl http://169.254.169.254/2008-09-01/meta-data/placement/availability-zone`

3. Prepare the Volume

First, attach the volume to a running EC2 instance. Make sure it’s at least the same type (i386 or x86_64) as the image you’re working on.

ec2-attach-volume vol-<your vol id> -i `curl http://169.254.169.254/2008-09-01/meta-data/instance-id` -d /dev/sdp

An EBS volume is a raw bit bucket. You need to partition it (if you’re in to that kind of thing) and create the filesystem on it. Partitions don’t really make sense here though, so just create a nice shiny filesystem on it once it’s mounted on the instance.

mke2fs -j /dev/sdp

In this instance I’m making an EXT3 filesystem, but you can use any filesystem that’s supported by the host machine. Please make sure though that the block device you specify (in this example /dev/sdp) matches what you told EC2 to mount your EBS volume on.

mkdir /ebs

mount /dev/sdp /ebs

This mounts your new filesystem on a directory called /ebs

mkdir /ebs/mnt

mkdir /ebs/download

mkdir /ebs/upload

mkdir /ebs/.ec2

This creates some handy directories to help you along with the process. This is what they do:

• mnt: Will be used as the mount point to access your image
• download: This is where you’ll to download your initial bundle to
• upload: When you bundle an image, put it here ready to be uploaded
• .ec2: This will contain your AWS access keys and your EC2 PEM files as follows:
  • s3.secret: S3 Secret Key
  • s3.access: S3 Access Key
  • ec2-pk.pem: EC2 Private Key
  • ec2-cert.pem: EC2 Certificate
  • id: EC2 user ID (Note: AWS account number, NOT Access Key ID)

ec2-download-bundle -b your-bucket -a `cat /ebs/.ec2/s3.access` -s `cat /ebs/.ec2/s3.secret` -k /ebs/.ec2/ec2-pk.pem -d /ebs/download -p your-image-name

This command pulls down the bundle you want to customise from S3. As I said before though, this will only work if you have sufficient rights on S3 to download the image and the EC2 private key that bundled it up and encrypted it in the first place.

ec2-unbundle -k /ebs/.ec2/ec2-pk.pem -s /ebs/download -d /ebs -m /ebs/download/your-image-name.manifest.xml

This command uncompresses and decrypts the image file from the downloaded bundle. It takes a while…

4. Customise

Now you’re ready to work. All you need is two shell scripts to go in your /ebs directory. work.sh mounts up your image (if it’s not mounted already) and chroot’s you in and you’re now up and running – customise to your hearts content. When you’re done, make sure you’ve logged out of all your work.sh scripts (yes you can run more than one) and then run bundle-and-upload.sh.

When you’re done, just shut down your host machine. When you want to work on it again later, just boot up a new AMI, attach your volume, mount it up and you’re at step 4 already.

Have fun…

The only criticism I have is that they don’t put enough emphasis on one of my key reasons for why it’s suddenly taken off. Cloud computing is not a new idea – it’s an extension of the Utility Computing that John McCarthy talked about in 1961. Although they only make a passing remark in section 3, I think one of the most important reasons it’s taken off is that the services Amazon provide were the first that were not a “solution looking for a problem”. Earlier offerings by the likes of Sun, HP and Intel all created a solution that they tried to sell to clients. The problem was that there were remarkably few problems that their solutions solved. Amazon simply exposed services that they were using internally already. That’s not to say the other reasons they give are not valid, I totally agree with them. I think they just missed a good point.

One of the topics I only glanced over is covered cover quite well in section 6 – Cloud Computing Economics. They provide some interesting example cost calculations. Although the numbers are obviously US centric, they do provide a nice way for a company to approach making the old “build vs buy” comparison.

In summary, I highly recommend this paper for anyone who wants to get the head around what this Cloud stuff is all about and what they need to do to prepare for it.

The Buildix image is now available for public use as ami-e4ca2f8d.

Here’s a couple of tips to keep in mind when rolling your own:

Remeber it’s Xen

I had problems getting my first batch of uploads working on the network. They would boot without a problem, but the networking would not initialize (as I could see from the console output). Things got a lot easier when I started manipulating the image locally using my own Xen installation. By booting it up on my local Xen server I could see where the problems were and fix them with a lot less pain and suffering.

The other benefit you get from testing your image in Xen is that you can bundle the image directly (ec2-bundle-image) instead of having to scrape a running machine (ec2-bundle-vol).

There can’t be only one

As the name implies, EC2 is all about having a big cloud of images. Chances are there will be a lot of instances of each image running out there. This means that the image itself needs to pick up its identity on boot up. Amazon provide a nice RESTful API that can provide each instance with information about itself. The documentation for the meta-data that’s available is available at http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/AESDG-chapter-instancedata.html. The stuff I needed the most was the public SSH key to allow to log in as root, and the hostname of the machine. Here’s what I added to /etc/rc.local to allow me to do this:

if [ ! -d /root/.ssh ] ; then
    mkdir -p /root/.ssh
    chmod 700 /root/.ssh
fi
# Fetch public key using HTTP
curl http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key > /tmp/my-key
if [ $? -eq 0 ] ; then
    cat /tmp/my-key >> /root/.ssh/authorized_keys
    chmod 600 /root/.ssh/authorized_keys
    rm /tmp/my-key
fi

hostname `curl http://169.254.169.254/latest/meta-data/hostname`

As always, there were also people who were interested in hearing about how it can be used for non Java projects. I had a good chat to one person who is interested in using it on a mixed Common LISP and Erlang project he’s working on. I’m looking forward to hearing how it goes for him. Due to lack of experience on my part I could unfortunately not help him much with the darcs problems he’s having though. Some people have all the fun…

It was also quite useful to speak to people about the problems they’re currently facing when trying to use CruiseControl. A common theme is people trying to manage large numbers of builds, or trying to build products across large numbers of different platforms. These are problems the dedicated ThoughtWorks development team are currently working on, so it’s great to get the validation that we’re putting effort into the things people care about now.

The whole point of Buildix is to help any Java based Agile Development Project get up and running as quickly as possible buy providing them with pre-configured and integrated version control system, continuous integration framework, wiki and issue tracking system. We chose our favourite products in each of these areas – Subversion, CruiseControl and Trac. Another common difficulty faced by our development teams, especially in the early stages of a project, is network access. Sometimes all we get from our clients when we arrive on site is a switch to allow us to get our laptops to talk to each other – no DNS, file shares, anything. Buildix can also help in situations like this, as it also runs Samba, and will run as a DNS and DHCP server if given the correct kernel boot parameters.

So, six months after we started, and after a few internal releases, we decided to give something back to the community that helps us do our job, and make Buildix available to everyone. Enjoy!