We started the project because at the time, setting up a new Continuous Integration server was quite an arduous task. The only real option out there for a Java project was CruiseControl, and it could take a new developer days to get their first build through the system. Thankfully though this is no longer the case.

Since then the whole CI landscape has changed. Just having a single “build server” is now more the exception than the rule. It’s all about build farms these days using tools like Cruise, Hudson and TeamCity. They integrate nicely with a variety of SCM’s and story tracking tools. Setting up a build environment with these tools is really easy now. I’d like to think that Buildix at least had something to do with helping people to see how easy it could be to get a CI environment up and running, and I know that at least in the case of Cruise this is true because I’ve been part of that team.

So – thank you to all of you who used Buildix and liked it and provided feedback. Thank you also to the current big players in the CI field who put effort into making sure that looking after your CI environment no longer needs to be a full time job for someone.

With this in mind it was decided to call the conference DevOpsDays. Videos of the talks will be online in the archives section soon, so I’ve decided to write down my thoughts about what went well and not so well – I am a fan of retrospectives.

What Went Well

The conference was in Ghent, a city in Belgium, and as expected had quite a heavy European contingent. It was not not exclusively so though, as Reductive Labs and Canonical both sent representatives over from the USA, and one of the speakers is an Aussie who just happened to be back packing around Europe at the time. One of my first great learnings though was that there are about 10 of us in the greater London area who are doing this kind of thing (2 of them ex-TWers) so we’ve started meeting regularly and exchanging ideas.

As there was such a large concentration of us around London, we’ve decided to try and subvert one of the London Geek Nights as a DevOps night to try and spread the word to more Devs. I’ll probably make more noise about it here when we’ve got dates confirmed.

The next great thing I loved was the format. It was split up into talks in the morning, with Open Spaces in the afternoon, over 2 days. This format actually worked very well as many of the open space sessions were inspired by the talks. Another interesting mix was that there was an even split between talks on tools and talks on practices.

I also discovered some great tools! Here’s a brief list of ones I liked the most:

• The Marionette Collective – using a message bus to administer 100′s of servers using pub/sub topics. You’ve got to watch the video to appreciate the full awesomeness.
• Cucumber-Nagios – writing tests to monitor your system using Cucumber, and then formatting the results as a Nagios plugin so you can easily wire it in to your monitoring system.
• Flapjack – another example of interesting use of a message bus. This time to parallelize the monitoring of 100′s of servers. I’m not sure I totally agree with the approach, but what really blew me away was they way they do graphing and reporting. It’s very impressive.
• collectd – collectd gathers statistics about the system it is running on and stores this information in RRD databases. One of the things I really like about this though is that there are plugins for it like Jcollectd which requires no code changes. Simply plug it into your classpath and it registers a JMX MBean that pumps data from your JVM to collectd (I think). I’ve not tried it in production yet though…

Overall the conference was very well received, and I’m looking forward to next years instalment. There’s a lot of talk about doing a North American version…

Not So Well

As we all know, the best bits about Away Days/Conferences are the talks in the hall between sessions. The problem with the program was that we barely had enough time for bio breaks between sessions, and everyone was quite burned out at the end of day two (not just because of the Belgian Beer). I think it’s a pretty easy thing to fix though and considering that it was the first conference of its type there’s bound to be some fine tuning required.

How many times have you come across a .jar or .dll file and wondered what version it is? Especially when using Open Source Software?

The most frightening one for me is when I’ve looked at a cluster of production servers and noticed that the .war file for the application deployed on it is a different size on one of the nodes. Which one was the correct one to deploy? Luckily this happened to me a long time ago, but I know that people out there are still having this problem today.

The solution is what I call “Self Identifying Software”. Every build of your software needs to have something that tells you what version it is and how to get back to the source code that created it. Having a build label or release number visible in your application is a good start, but it does not make your software Self Identifying. Product companies have been doing this for ever. The problem is that for that number to be useful (particularly when you’re trying to access the source code to reproduce and fix a bug) you then need to refer to a build system or release notes to find out where the source code came from (if you’re lucky). It often also does not apply to development builds. To be truly Self Identifying you need to make sure that every build (including builds developers create on their workstations) also includes enough information from the SCM system so that anyone who has access to the source code can go right back to the exact source code that produced that binary. For example, if you use Subversion as your SCM then this will be a URL and a revision number.

This is not exactly a new concept, it’s something I (and others) have been doing for a number of years now. The reason I’ve decided to write about it now though is that recently I was showing a new guy around one of the projects I’m working on at the moment, and when I showed him how to determine which version of the app was deployed he was delighted.

How do I get my software to identify itself?

There are generally two scenarios where I would want to get the identity of some software. The first is where I have a service or system running somewhere that I would like to know the version of, and the second is where I have a binary artefact and I’m trying to determine it’s provenance.

When querying a service, the easiest is to be able to hit a well known URL (or make a well known service call) that tells me what I need to know. What the exact URL or service call is is not important, as long as those who need it know how to invoke it. This is most useful then you don’t actually have access to the binaries that power the service.

However, if you do have access to the binaries then it’s usually easiest just to examine the meta data of the file to see where it came from. Most modern binary formats allow for this in some way. If you’re working with Java then all .jar, .war and .ear files allow you to put this into the META-INF/MANIFEST.MF file. Creative use of versioninfo resources will allow you to achieve similar results if you’re dealing with Windows. Note that we’re not talking about putting this information into the file name itself – file names change.

How do I use it?

Having Self Identifying Software goes beyond just being able to quickly track down the source code that built the binaries you’re dealing with. Here are a few other uses I’ve found:

• When used as part of a Continuous Deployment system you can verify that you’ve got the correct version of the binaries before you start deploying things, and you can then use the service call to verify that the correct version of the app is up and running at the end of the deployment processes.
• When used as part of a dynamic system dashboard you can quickly and easily see what’s installed where instead of relying on spreadsheets or documents that often get forgotten.

If you find any more, please let me know…

“Today’s so-called cloud isn’t really a cloud at all. It’s a bunch of corporate dirigibles painted to look like clouds. You can tell they’re fake because they all have logos on them. Real clouds don’t have logos.”

As much as I enjoyed reading their paper, I must say that I disagree with a lot of what the guys at Maya are saying. Yes, there is a large dose of spin and hand wavey magic going on with the current leading fashion trend (that bit is totally true). Yes, it is very easy to tightly couple your application to a cloud vendor. The thing is though that it’s not that different from the tie in you get when selecting what language you use to develop your application in, which third party libraries you use or even what operating system(s) to target. The only real difference I can think of is that if for some reason the cloud vendor you’re backing stops running your app goes down, unlike all those mission critical OS/2 applications that are still running out there…

I’m pretty sure that you’d have fair warning before the plug was pulled though, especially if you’re still paying them money every month for their services.

The real questions you need to ask before doing anything on any cloud service are:

1. What is the problem I’m trying to solve?
2. Do any of the cloudy offerings actually help me solve that problem?
3. What is the cost difference between deploying this app in the cloud vs our own infrastructure (assuming you have any of your own)?
4. What is the point at which that will change? Is there a usage point where it would be cheaper for me to move off the cloud?
5. If I chose platform X, how hard will it be to move to platform Y?

Cloud is not a magic silver bullet – such things don’t exist. As with any technology choice you make, you need to select the most cost effective one for the problem you have, and try your hardest to ignore the FUD.

The only criticism I have is that they don’t put enough emphasis on one of my key reasons for why it’s suddenly taken off. Cloud computing is not a new idea – it’s an extension of the Utility Computing that John McCarthy talked about in 1961. Although they only make a passing remark in section 3, I think one of the most important reasons it’s taken off is that the services Amazon provide were the first that were not a “solution looking for a problem”. Earlier offerings by the likes of Sun, HP and Intel all created a solution that they tried to sell to clients. The problem was that there were remarkably few problems that their solutions solved. Amazon simply exposed services that they were using internally already. That’s not to say the other reasons they give are not valid, I totally agree with them. I think they just missed a good point.

One of the topics I only glanced over is covered cover quite well in section 6 – Cloud Computing Economics. They provide some interesting example cost calculations. Although the numbers are obviously US centric, they do provide a nice way for a company to approach making the old “build vs buy” comparison.

In summary, I highly recommend this paper for anyone who wants to get the head around what this Cloud stuff is all about and what they need to do to prepare for it.

This is always a hard one to answer. In the last place I worked as a System Administrator we used Split-Horizon DNS as one of the ways to solve connecting to the correct host for a service. Our applications had hard coded hostnames for key services such as db.internal.domain, loghost.internal.domain and mailhost.internal.domain. Depending on which environment they ran in, when the did a DNS lookup they would get an answer that pointed them to the correct server providing that service for the environment they were physically running in. Combine this with tools like an LDAP Server in each of these environments containing the rest of the system configuration and life is easy. Or is it?

DNS and LDAP servers are relatively straight forward for experienced Sys Admins to understand and maintain, but they still can’t be described as easy to use, which is probably why so few people do it. This limits our options for fixing things down to two general categories – fix DNS and LDAP admin tools so they are easy to use, or find another way to solve the problem.

We chose the path of least resistance and went for the second option – creating ESCAPE. In the Unix tradition, it’s an app that takes one problem and solves it simply and well (at least in our opinion). It provides a RESTful interface for both the setting and getting of environment configuration. GET requests will retrieve configuration, whereas POST/PUT requests will create new entries or update them.

The URL scheme we’ve decided on is most easily demonstrated by constructing an example URL:

As the tool is now usable we’ve happily released version 0.1. Currently we’re busy putting a lot of work into making the management interface understandable and intuitive to use. The closest thing we currently have to a roadmap is our ToDo list.

Please join us on the discussion group with your feedback.

As I’m normally brought in with the developers, I’ve been trying to find efficient ways to engage with the client Operations teams. Normally I end up having one on one conversations with various members of the team, try to find out what their current processes are, what their major challenges are and what their concerns are regarding the project I’m involved in. I usually do this to keep the safety levels high. The problem though is that it takes quite a lot of time and effort to get things going and get some momentum going.

At my current client though I didn’t have the time or access to the people to do things the normal way. A meeting was arranged with the key Operations stake holders and I effectively had just 2 hours to explain our development process in general, and Continuous Integration and Build Pipelines in detail. While talking with Graham Brooks about what we wanted to cover, he came up with the idea of running it as a mini retrospective.

After the usual introductions, we gave them 15 minutes to list the Good, the Bad and the Puzzles of their current development and release process. We had good participation from the group and as expected had a high number Bad entries. After talking through the cards and grouping them into related sections, we then allowed them to vote on the ones they most wanted to talk about. Most votes went to the core pain points, and we spent the rest of the time talking about how our process would address those issues. It also helped a lot that most of the Good entries related to the automation they already have in place…

By the end of the meeting no was talking about the bad old days (lobbing releases over the wall). Everyone was engaged starting to get some spirit of collective ownership going in the whole delivery process and that breaking down the walls that exist between the various silos was high on the list of things to do. Rather than talk to them about our process and how we would like to interact with them, we had allowed them to lead the discussion on which elements from our toolbox would have the greatest value for them.

All I need to do now is learn how to be as good a facilitator as Graham was…

The list of available products out there has grown a lot in the past 18 months, and the features that they support are really great. Since I did the last review I’ve actively avoided having a look at the other tools out there to keep a clear focus on what I wanted to see in Cruise. Doing this review has been a great way for me to see what everyone else has been up to.

Just having loads of features does not automatically make for a good tool though. Instead of having a shooting match between who does what, I’ve taken a little sample Java servlet that I use for demos and tried to get it working with all the tools. This project is hosted on a local subversion repository. I’m going to try set it up to simply run unit tests and create my distributable .war file. Areas that I’m going to look at are:

• Installation (on Linux, OSX and Windows)
• Setting up my existing project
• What did the tool inspire me to try next

Tools that I’ll be trying out are (in alphabetical order):

• AntHillPro 3.5.3
• Bamboo 2.1.1
• Cruise 1.0
• CruiseControl 2.7.3
• Hudson 1.252
• Pulse 1.2.56
• TeamCity 3.1.2

AntHillPro

Installers

There are no native platform installers, but you can choose from 3 packages. Each package contains both the server and the agent components.

• Graphical Java Installer
• Windows Command Line Installer
• Unix Command Line Installer

Installation

I chose the Unix Command Line. Unpacked the tarball and ran the script. It’s unique in that all the script does is kick off an ant build that asks you questions and does the installation. The defaults all seemed pretty reasonable. Installing an agent is done separately and follows the same process. Once you start the processes, hit the web frontend and finish the setup.

Setting up my project

This is where things go interesting. There was no obvious “add your project here” button. A bit of clicking around, and I found something under the Admin tab that looked like a good place to start. Then it asked me if I want a Life-Cycle based project or a Non Life-Cycle based one? What’s a Life-Cycle in this context? I fuddled around and took random guesses, and then figured out that I needed to add a workflow to my project to actually do anything. Fair enough, only I was half way through configuring my new workflow when it asked for my repository, but only gave me a dropdown list to choose from. Back out, into the admin page, find out where to add new repositories, add a Subversion one. Once again try to finish configuring my new project (at least it remembered where I was), but then it all fell apart when trying to add the repository to the workflow. Kept on throwing up “AttributeMissingException: project is missing”. Decided that this had already taken up far too much of my time just to get to this point, and so I gave up.

Bamboo

Installers

A single installer that contains both the server and the agent.

• Windows .exe installer
• OSX .dmg file
• tarball

Installation

I picked the “Linux” tarball, even though I’m testing on OSX. I prefer doing this when evaluating products as I think it gives me a better feel for what’s actually going to be installed, and I’m guaranteed I can get rid of it efficiently and easily when I’m done playing with it. Unpackd the tarball, ran the start script. All looks good, but when you try use it it asks you to edit a file in the webapp path pointing to the installation directory. Nasty. Once that’s fixed and started again, you’re asked for your license key. There’s a handy link at the bottom to go get your key, copy paste and you’re away through the rest of the wizard.

Setting up my project

Clicked the Create Plan link. Still has my pet hate of wanting you to specify UPPER CASE KEYS for everything. Are we still using ancient Big Blue iron here that has such limitations? Wants me to specify an ant target which is also a little weird – I just want the default target! Other than that all very nice. I like how it goes right into the activity page once you’ve set up your project and you can see what’s going on, including SVN checkout.

I then wanted to play with

Agents. Installed agents on Ubuntu and WinXP (server running on OSX). It’s pretty easy really. Agent matrix shows you which projects will build on which agents. Also went to look at all the pretty graphs, but I don’t have much build history right now. Bamboo has built in Clover integration, but I use cobertura in my project for coverage. It looks like they have some interesting reports around code coverage, but I can’t run them…

Cruise

Disclosure – I wrote the installers for Cruise, so obviously I like them.

Installers

Separate installers for the server and the agent components.

• Windows .exe installer
• OSX .app packaged in a zip
• Linux RPM and Debian packages

Installation

This is one of the few products that does not ship a generic tarball or zip file. As I’m doing all this on a Mac, I pulled down the server and agent zip files, unpacked them, and dragged the .app’s over to my Applications folder. Double click on each and we’re up. Once the server is up it opens up the admin interface in your browser. The first thing you’re asked for is your license key.

Setting up my project

The very first thing you’re hit with (after you’ve put in your license key) is a wizard to set up your project. Follow the prompts, and fill in where your test results and artefacts are stored. It won’t let you create the project until it verifies that it can connect to your source repository, so make sure it’s up before you start.

I then wanted to play with

Pipelines. I presented one of the very first papers on Build Pipelines with Jez Humble (now product manager for Cruise) and Dan North at Agile 2006. When I first started implementing them I had to do all kinds of unholy hacks with CruiseControl, but the benefit of using CI like this was well worth the pain at the time. At the moment you have to jump into nasty XML to configure pipelines in Cruise, but the whole process should be wizarded up in the next release. Having lots of agents allows you to split up jobs in your pipeline stages quite nicely so they run in parallel and speed up your feedback loop.

The other great feature I like that product companies are always asking me about is the ability to run parallel builds for a stage on multiple platforms. Whether you’ve got a .Net app that you need to test across multiple versions of Windows, or a web app that needs to be tested from a whole matrix of different web browsers, Cruise makes it easy to run the same tests in all these different environments using the same build of your application. If the tests pass on all of them, your stage will pass, and you know that your product is playing nicely with all of them.

CruiseControl

Installers

• Windows .exe installer
• Generic zip file

Installation

More unzip and run script. It just comes up with the sample project.

Setting up my project

This is where the venerable Granddaddy of CI servers really started showing it’s age. I had to manually check out my project, then edit the xml config file to get it going. If I’d not been using it for almost 4 years I’d get pissed off and move along.

I then wanted to play with

Something else. If you’re using it already and it’s doing the job you want it to, then there is no real reason to change. If you’re new to CI and are looking for something easy to get you started, then I’m sad to say I suggest you look elsewhere…

Hudson

Installers

• A single war file for everyone.

Installation

The most basic of all, just run java -jar hudson.war or drop the war file into your favourite servlet container. Makes it kinda hard to know what version you have though, and by the look if it they release new versions quite regularly.

Setting up my project

Not too bad. Pointing at unit test and artefacts was a bit weird as it starts one level higher than I expected. I liked the weather reports. Because the build was failing (because it could not publish any artefacts or find test results), it had little storm clouds. Nice.

I then wanted to play with

The plugins. There’s quite a few, and I like how you can just download them and go.

Pulse

Installers

• Windows .exe installer
• tarball
• zip

Installation

Another unpack the tarball and run the start script. I see a trend here. Hit the web interface and it asks you for a data directory. Also needs a license key to get going, but has a nice link there to go get one.

Setting up my project

Nice start screen with list of things to do. Hit add project, then got confused about what the URL was it was asking for. Got a test button for your SCM – yay! Setting up the artefacts you want is easy, it all just works. Only thing I can moan about is that I can’t see what’s going on in the builds, and it feels slower than the rest for some reason.

I then wanted to play with

Personal builds! It’s not something I’ve ever played around with, but it looked kinda straight forward with pulse, and it is! Just install a little tool on your development workstation and use it to invoke a personal build. Does require a bit of extra data in the project, but it’s great.

It also has agents. I love agents. They have an interesting way of doing them though. You point the agent at the server. I suppose that’s one way of getting around the security implications of other people getting their hands on your code. Agents auto-discover what their hosts provide, and when configuring a project you get a drop down list of what’s provided already. You’ve still got the freedom to add your own stuff too.

It has stages, kinda like our build pipelines. To use them nicely though you need to start editing their XML, but they’ve got great help along the right for you to see what your options are. Their stages run in parallel, so it’s not really like our Cruise pipelines, but at least they got the parallel stuff going.

Team City

Installers

• Windows .exe installer
• tarball

Installation

Unpack the tarball, find and run the start script.

Setting up my project

Hit the Create Project button. Interesting that you create the project first, then add the config after. Test SCM button. Auto tagging and code coverage built in. Nice. Also agent based. Option to automagically run emma for you. Tried it, but because I already use cobertura it barfed. It lost my artefacts config somehow. Lots of options. Also by default does not trigger a build on check in – what’s with that?

I then wanted to play with

Integration with IntelliJ. I’m a big fan of IntelliJ and use it a lot. It makes sense in a developer centric world, because the IDE is the developers portal into the world of code. I like the way you can click on a failed test and it takes you to the test code, as well as the ability to take responsibility for fixing a broken build.

Summary

The world of CI servers has come a long way in the past 18 months. It’s certainly reinforced my belief that competition is a great driver for innovation. When I started at ThoughtWorks clients would often complain that they’d like to do Continuous Integration, but setting up a CI server was way too hard. Complexity of tool configuration is now no longer an acceptable excuse for not doing CI.

It’s also great to see the trend of agents being so popular. The days of having a big mother of a build server for a project or small group of teams is over. After one of the Cruise demos I did here in the UK our client commented that it was great that he could finally find a use for all the overpowered desktop machines their managers have – use them as build agents!

As a final note for those of you who are trying to decide on what CI server to use – remember to pick one that supports your process and what you want CI to do for you.

Last year at the conference I got a chance to show off the new UI we did for CruiseControl with Erik Dörnenburg. Afterwards I was having a chat with Martin Fowler, and I commented on how I’d like to see a whole track at JAOO dedicated to Building and Deploying software. While I’m sure I can’t take all the credit for it, this year there is going to be a short Build track with me talking about Continous Integration. Hope to see you there…