I’ve been putting off this post for a few months now, but I think the time has finally come to admit what I’m sure people who care have guessed for a while – active development on Buildix has stopped, and will probably not resume. The site will stay up for the foreseeable future, nothing will vanish, but nothing new will be added either.

We started the project because at the time, setting up a new Continuous Integration server was quite an arduous task. The only real option out there for a Java project was CruiseControl, and it could take a new developer days to get their first build through the system. Thankfully though this is no longer the case.

Since then the whole CI landscape has changed. Just having a single “build server” is now more the exception than the rule. It’s all about build farms these days using tools like Cruise, Hudson and TeamCity. They integrate nicely with a variety of SCM’s and story tracking tools. Setting up a build environment with these tools is really easy now. I’d like to think that Buildix at least had something to do with helping people to see how easy it could be to get a CI environment up and running, and I know that at least in the case of Cruise this is true because I’ve been part of that team.

So – thank you to all of you who used Buildix and liked it and provided feedback. Thank you also to the current big players in the CI field who put effort into making sure that looking after your CI environment no longer needs to be a full time job for someone.

A few months ago I got an email from Patrick Debois who I’d met at CITCON Europe asking if I’d be interested in speaking at the first conference aimed at System Administrators practising/interested in/sceptical about Agile. One of the key beliefs of those of us doing this already is that Agile practices are generally too narrowly focussed in their implementation. At the moment it’s primarily the Development organization who drive its adoption, but to get the most benefit Development and Operations groups within an organization need to work together.

With this in mind it was decided to call the conference DevOpsDays. Videos of the talks will be online in the archives section soon, so I’ve decided to write down my thoughts about what went well and not so well – I am a fan of retrospectives.

What Went Well

The conference was in Ghent, a city in Belgium, and as expected had quite a heavy European contingent. It was not not exclusively so though, as Reductive Labs and Canonical both sent representatives over from the USA, and one of the speakers is an Aussie who just happened to be back packing around Europe at the time. One of my first great learnings though was that there are about 10 of us in the greater London area who are doing this kind of thing (2 of them ex-TWers) so we’ve started meeting regularly and exchanging ideas.

As there was such a large concentration of us around London, we’ve decided to try and subvert one of the London Geek Nights as a DevOps night to try and spread the word to more Devs. I’ll probably make more noise about it here when we’ve got dates confirmed.

The next great thing I loved was the format. It was split up into talks in the morning, with Open Spaces in the afternoon, over 2 days. This format actually worked very well as many of the open space sessions were inspired by the talks. Another interesting mix was that there was an even split between talks on tools and talks on practices.

I also discovered some great tools! Here’s a brief list of ones I liked the most:

• The Marionette Collective – using a message bus to administer 100’s of servers using pub/sub topics. You’ve got to watch the video to appreciate the full awesomeness.
• Cucumber-Nagios – writing tests to monitor your system using Cucumber, and then formatting the results as a Nagios plugin so you can easily wire it in to your monitoring system.
• Flapjack – another example of interesting use of a message bus. This time to parallelize the monitoring of 100’s of servers. I’m not sure I totally agree with the approach, but what really blew me away was they way they do graphing and reporting. It’s very impressive.
• collectd – collectd gathers statistics about the system it is running on and stores this information in RRD databases. One of the things I really like about this though is that there are plugins for it like Jcollectd which requires no code changes. Simply plug it into your classpath and it registers a JMX MBean that pumps data from your JVM to collectd (I think). I’ve not tried it in production yet though…

Overall the conference was very well received, and I’m looking forward to next years instalment. There’s a lot of talk about doing a North American version…

Not So Well

As we all know, the best bits about Away Days/Conferences are the talks in the hall between sessions. The problem with the program was that we barely had enough time for bio breaks between sessions, and everyone was quite burned out at the end of day two (not just because of the Belgian Beer). I think it’s a pretty easy thing to fix though and considering that it was the first conference of its type there’s bound to be some fine tuning required.

How often has someone come up to you and asked you what build of your software is currently deployed in a specific environment?

How many times have you come across a .jar or .dll file and wondered what version it is? Especially when using Open Source Software?

The most frightening one for me is when I’ve looked at a cluster of production servers and noticed that the .war file for the application deployed on it is a different size on one of the nodes. Which one was the correct one to deploy? Luckily this happened to me a long time ago, but I know that people out there are still having this problem today.

The solution is what I call “Self Identifying Software”. Every build of your software needs to have something that tells you what version it is and how to get back to the source code that created it. Having a build label or release number visible in your application is a good start, but it does not make your software Self Identifying. Product companies have been doing this for ever. The problem is that for that number to be useful (particularly when you’re trying to access the source code to reproduce and fix a bug) you then need to refer to a build system or release notes to find out where the source code came from (if you’re lucky). It often also does not apply to development builds. To be truly Self Identifying you need to make sure that every build (including builds developers create on their workstations) also includes enough information from the SCM system so that anyone who has access to the source code can go right back to the exact source code that produced that binary. For example, if you use Subversion as your SCM then this will be a URL and a revision number.

This is not exactly a new concept, it’s something I (and others) have been doing for a number of years now. The reason I’ve decided to write about it now though is that recently I was showing a new guy around one of the projects I’m working on at the moment, and when I showed him how to determine which version of the app was deployed he was delighted.

How do I get my software to identify itself?

There are generally two scenarios where I would want to get the identity of some software. The first is where I have a service or system running somewhere that I would like to know the version of, and the second is where I have a binary artefact and I’m trying to determine it’s provenance.

When querying a service, the easiest is to be able to hit a well known URL (or make a well known service call) that tells me what I need to know. What the exact URL or service call is is not important, as long as those who need it know how to invoke it. This is most useful then you don’t actually have access to the binaries that power the service.

However, if you do have access to the binaries then it’s usually easiest just to examine the meta data of the file to see where it came from. Most modern binary formats allow for this in some way. If you’re working with Java then all .jar, .war and .ear files allow you to put this into the META-INF/MANIFEST.MF file. Creative use of versioninfo resources will allow you to achieve similar results if you’re dealing with Windows. Note that we’re not talking about putting this information into the file name itself – file names change.

How do I use it?

Having Self Identifying Software goes beyond just being able to quickly track down the source code that built the binaries you’re dealing with. Here are a few other uses I’ve found:

• When used as part of a Continuous Deployment system you can verify that you’ve got the correct version of the binaries before you start deploying things, and you can then use the service call to verify that the correct version of the app is up and running at the end of the deployment processes.
• When used as part of a dynamic system dashboard you can quickly and easily see what’s installed where instead of relying on spreadsheets or documents that often get forgotten.

If you find any more, please let me know…

I’ve been doing even more reading than normal lately on the subject of Clouds lately as quite a few of us within ThoughtWorks who are going to be speaking on the subject next month are comparing notes. It follows that when I clicked the link for “The Wrong Cloud” I was not actually prepared for a delightfully entertaining paper that would contain my quote of the day:

“Today’s so-called cloud isn’t really a cloud at all. It’s a bunch of corporate dirigibles painted to look like clouds. You can tell they’re fake because they all have logos on them. Real clouds don’t have logos.”

As much as I enjoyed reading their paper, I must say that I disagree with a lot of what the guys at Maya are saying. Yes, there is a large dose of spin and hand wavey magic going on with the current leading fashion trend (that bit is totally true). Yes, it is very easy to tightly couple your application to a cloud vendor. The thing is though that it’s not that different from the tie in you get when selecting what language you use to develop your application in, which third party libraries you use or even what operating system(s) to target. The only real difference I can think of is that if for some reason the cloud vendor you’re backing stops running your app goes down, unlike all those mission critical OS/2 applications that are still running out there…

I’m pretty sure that you’d have fair warning before the plug was pulled though, especially if you’re still paying them money every month for their services.

The real questions you need to ask before doing anything on any cloud service are:

1. What is the problem I’m trying to solve?
2. Do any of the cloudy offerings actually help me solve that problem?
3. What is the cost difference between deploying this app in the cloud vs our own infrastructure (assuming you have any of your own)?
4. What is the point at which that will change? Is there a usage point where it would be cheaper for me to move off the cloud?
5. If I chose platform X, how hard will it be to move to platform Y?

Cloud is not a magic silver bullet – such things don’t exist. As with any technology choice you make, you need to select the most cost effective one for the problem you have, and try your hardest to ignore the FUD.

Last year at JAOO I had the chance to speak to Markus from Software Engineering Radio about the talk I gave there on Continuous Integration. It’s finally available now over here. The slides that go along with the talk are available from the JAOO site.

Ola Bini and myself are going to be talking about Java in the Google Cloud at Skills Matter in London on 11 May.

Registration and more info at http://skillsmatter.com/podcast/ajax-ria/java-in-the-google-cloud

It’s been a long time since my first post on EC2 AMI Creation Tips. At the time the primary images people were using were the RedHat based ones supplied by Amazon, but I was trying to do something Ubuntu based. Since then a whole host of other well prepared images are now available. I was even lucky enough to be invited to create an AMI for Sun’s launch of OpenSolaris on EC2, but am not allowed to say much more about it…

Recently though I’ve been speaking to more and more people who are trying to take an existing AMI and customise it for their own use. They do this by booting the AMI they want to base theirs on, doing the customization, then bundling up that volume. Generally they do pretty well, but there are three common themes that crop up that often cause pain: transient runtime configuration being bundled up, the time (and to a lesser extent effort) it takes to bundle the new image in the first place and making further changes to the image down the line.

Thankfully there is a simple single solution to these three problems – bundle from an image, not a running volume, and keep that image (or a set of images) along with some nice helper scripts on an EBS volume. That’s the theory, but as always there’s something in the real world that stops it being easy. By default, only the owner of an image can download and unpack an image directly from S3 and the images are encrypted with the owners EC2 private key. For this process to work, you’ll need to at least bootstrap yourself initially by going through the well known and well documented process of bundling a running system. After that though it’s easy. Really. Promise…

Let’s have a closer look at the problems we’re trying to solve first though before I go into how we fix them.

Transient Runtime Pollution

The most frustrating of these is the udev system flagging the source machines MAC address for eth0 and so making their custom image unusable because the network interface does not come up. There are still distributions out there which try to be “helpful” by remembering which physical device, such as network card, maps to which logical device name, such as eth0. This is not in itself a bad thing. This is one feature I was crying for 10 years ago when I started using Linux on bigger iron. I would dread adding another network card to a server because I would normally end up having to re-label the external interfaces. The thing is though that you’re now creating an image that could be running anywhere and you don’t have physical or even console access to it.

Other examples of things that break are helper scripts. Because we’re now on an operating system image that is meant to be able to run anywhere, there are certain things you want to run only once the very first time the system boots. Once they’ve run the these scripts either create a lock file, clear their own executable bits or even delete themselves. If you’re trying to re-bundle and image you’ve already booted, you need to make sure you back out these changes.

Doing your customization in an image that has never actually been booted helps you keep all these things pristine.

Time and Effort of Creation

This one is actually quite straight forward. When you’re bundling a running volume, what happens under the hood is:

1. A new sparse file for the image is created
2. A new filesystem is created on the new image file
3. This new filesystem is mounted somewhere
4. The contents of your running volume is copied into the new image file
5. The new filesystem is then unmounted
6. The image file is compressed and encrypted
7. The compressed and encrypted file is then split into chunks
8. Your manifest is created

Some of these steps very I/O intensive. When you’re working with an image though, steps 1 to 5 don’t happen (well, steps 3 and 5 are needed for you to make changes) so you’ll be doing almost 50% less IO. This means that bundling a new image will take about half time. If you work with your image on an EBS volume it’ll be even faster as they have better performance characteristics than the standard instance stores.

Bundling and uploading images are not simple commands though. You need to specify things like your AWS access key and provide your EC2 encryption key. There’s options for which kernels and ramdisks to use. There’s lots of typing which means lots of room for human error. The way to get around this is to have small shell scripts with all these options in them. Now they are simple commands…

Maintenance

Once you’ve got your new AMI looking the way you want and doing the things you need, chances are that a few weeks after you’ve started using it you find that there’s a security fix or package update you’d like to apply. Often this ends up with people starting the whole process from scratch again. Boot up a new instance of the AMI you want to update, update it, type in all those commands and remember the options you used to bundle the volume and upload the new one. If you kept your scripts and image on an EBS you could simply attach it to a running instance and make the fixes there using the same scripts you used last time. Hows that for repeatability?

“So, just how do I work with an image then?” I hear you ask. Here’s a basic outline to get you started.

1. Set up your environment

These steps assume that you have the EC2 AMI and API tools installed locally, and that you’re running the commands on an EC2 instance. If you don’t have them, please look at EC2 AMI Tools and EC2 API Tools.

You also need some environment variables configured to make life easier:

EC2_PRIVATE_KEY=/path/to/your/EC2/private/key
EC2_CERT=/path/to/your/EC2/cert

2. Create your EBS Volume

The hardest thing will be working out how big you need to make it. Absolute worst case will be 20gb per image, but in reality 10gb should be plenty. Remember though that an EBS can only be mounted in the availability zone it is created in, so this command creates one in the same zone you are in.

ec2-create-volume -s 10 -z `curl http://169.254.169.254/2008-09-01/meta-data/placement/availability-zone`

3. Prepare the Volume

First, attach the volume to a running EC2 instance. Make sure it’s at least the same type (i386 or x86_64) as the image you’re working on.

ec2-attach-volume vol-<your vol id> -i `curl http://169.254.169.254/2008-09-01/meta-data/instance-id` -d /dev/sdp

An EBS volume is a raw bit bucket. You need to partition it (if you’re in to that kind of thing) and create the filesystem on it. Partitions don’t really make sense here though, so just create a nice shiny filesystem on it once it’s mounted on the instance.

mke2fs -j /dev/sdp

In this instance I’m making an EXT3 filesystem, but you can use any filesystem that’s supported by the host machine. Please make sure though that the block device you specify (in this example /dev/sdp) matches what you told EC2 to mount your EBS volume on.

mkdir /ebs

mount /dev/sdp /ebs

This mounts your new filesystem on a directory called /ebs

mkdir /ebs/mnt

mkdir /ebs/download

mkdir /ebs/upload

mkdir /ebs/.ec2

This creates some handy directories to help you along with the process. This is what they do:

• mnt: Will be used as the mount point to access your image
• download: This is where you’ll to download your initial bundle to
• upload: When you bundle an image, put it here ready to be uploaded
• .ec2: This will contain your AWS access keys and your EC2 PEM files as follows:
  • s3.secret: S3 Secret Key
  • s3.access: S3 Access Key
  • ec2-pk.pem: EC2 Private Key
  • ec2-cert.pem: EC2 Certificate
  • id: EC2 user ID (Note: AWS account number, NOT Access Key ID)

ec2-download-bundle -b your-bucket -a `cat /ebs/.ec2/s3.access` -s `cat /ebs/.ec2/s3.secret` -k /ebs/.ec2/ec2-pk.pem -d /ebs/download -p your-image-name

This command pulls down the bundle you want to customise from S3. As I said before though, this will only work if you have sufficient rights on S3 to download the image and the EC2 private key that bundled it up and encrypted it in the first place.

ec2-unbundle -k /ebs/.ec2/ec2-pk.pem -s /ebs/download -d /ebs -m /ebs/download/your-image-name.manifest.xml

This command uncompresses and decrypts the image file from the downloaded bundle. It takes a while…

4. Customise

Now you’re ready to work. All you need is two shell scripts to go in your /ebs directory. work.sh mounts up your image (if it’s not mounted already) and chroot’s you in and you’re now up and running – customise to your hearts content. When you’re done, make sure you’ve logged out of all your work.sh scripts (yes you can run more than one) and then run bundle-and-upload.sh.

When you’re done, just shut down your host machine. When you want to work on it again later, just boot up a new AMI, attach your volume, mount it up and you’re at step 4 already.

Have fun…

I found a link to Above the Clouds, a paper on Cloud Computing recently published by a quartet of UC Berkeley RAD Lab professors. I’ve been quite disappointed with publications on the subject of the latest buzzword taking the world by storm right now, so I was not expecting much when I first clicked on the link. The thing is, as I started reading through the Executive Summary it all sounded very familiar. The outline the give in the summary follows the same outline as a talk I gave in November last year at the ThoughtWorks London office for the London Java Community.

The only criticism I have is that they don’t put enough emphasis on one of my key reasons for why it’s suddenly taken off. Cloud computing is not a new idea – it’s an extension of the Utility Computing that John McCarthy talked about in 1961. Although they only make a passing remark in section 3, I think one of the most important reasons it’s taken off is that the services Amazon provide were the first that were not a “solution looking for a problem”. Earlier offerings by the likes of Sun, HP and Intel all created a solution that they tried to sell to clients. The problem was that there were remarkably few problems that their solutions solved. Amazon simply exposed services that they were using internally already. That’s not to say the other reasons they give are not valid, I totally agree with them. I think they just missed a good point.

One of the topics I only glanced over is covered cover quite well in section 6 – Cloud Computing Economics. They provide some interesting example cost calculations. Although the numbers are obviously US centric, they do provide a nice way for a company to approach making the old “build vs buy” comparison.

In summary, I highly recommend this paper for anyone who wants to get the head around what this Cloud stuff is all about and what they need to do to prepare for it.

Whenever I talk to new clients about Continuous Integration, and especially about using Build Pipelines to extend their CI process to cover testing things like the deployment of their application, there is always one question that is guaranteed to pop up – how do I manage the configuration of my application in all these environments?

This is always a hard one to answer. In the last place I worked as a System Administrator we used Split-Horizon DNS as one of the ways to solve connecting to the correct host for a service. Our applications had hard coded hostnames for key services such as db.internal.domain, loghost.internal.domain and mailhost.internal.domain. Depending on which environment they ran in, when the did a DNS lookup they would get an answer that pointed them to the correct server providing that service for the environment they were physically running in. Combine this with tools like an LDAP Server in each of these environments containing the rest of the system configuration and life is easy. Or is it?

DNS and LDAP servers are relatively straight forward for experienced Sys Admins to understand and maintain, but they still can’t be described as easy to use, which is probably why so few people do it. This limits our options for fixing things down to two general categories – fix DNS and LDAP admin tools so they are easy to use, or find another way to solve the problem.

We chose the path of least resistance and went for the second option – creating ESCAPE. In the Unix tradition, it’s an app that takes one problem and solves it simply and well (at least in our opinion). It provides a RESTful interface for both the setting and getting of environment configuration. GET requests will retrieve configuration, whereas POST/PUT requests will create new entries or update them.

The URL scheme we’ve decided on is most easily demonstrated by constructing an example URL:

As the tool is now usable we’ve happily released version 0.1. Currently we’re busy putting a lot of work into making the management interface understandable and intuitive to use. The closest thing we currently have to a roadmap is our ToDo list.

Please join us on the discussion group with your feedback.

I like to describe my job at ThoughtWorks as helping Developers and Operations realize that they’re playing on the same team. No matter how awesome your code is, how elegantly you’ve solved the problem at hand, how nice and readable the code is – if you can’t get it into production your software is just a collection of bits. Likewise, you can have the best network, the most scalable hardware, the neatest cable patching scheme – but it’s just a big fancy heater if it’s not running the code your business needs.

As I’m normally brought in with the developers, I’ve been trying to find efficient ways to engage with the client Operations teams. Normally I end up having one on one conversations with various members of the team, try to find out what their current processes are, what their major challenges are and what their concerns are regarding the project I’m involved in. I usually do this to keep the safety levels high. The problem though is that it takes quite a lot of time and effort to get things going and get some momentum going.

At my current client though I didn’t have the time or access to the people to do things the normal way. A meeting was arranged with the key Operations stake holders and I effectively had just 2 hours to explain our development process in general, and Continuous Integration and Build Pipelines in detail. While talking with Graham Brooks about what we wanted to cover, he came up with the idea of running it as a mini retrospective.

After the usual introductions, we gave them 15 minutes to list the Good, the Bad and the Puzzles of their current development and release process. We had good participation from the group and as expected had a high number Bad entries. After talking through the cards and grouping them into related sections, we then allowed them to vote on the ones they most wanted to talk about. Most votes went to the core pain points, and we spent the rest of the time talking about how our process would address those issues. It also helped a lot that most of the Good entries related to the automation they already have in place…

By the end of the meeting no was talking about the bad old days (lobbing releases over the wall). Everyone was engaged starting to get some spirit of collective ownership going in the whole delivery process and that breaking down the walls that exist between the various silos was high on the list of things to do. Rather than talk to them about our process and how we would like to interact with them, we had allowed them to lead the discussion on which elements from our toolbox would have the greatest value for them.

All I need to do now is learn how to be as good a facilitator as Graham was…