With this in mind it was decided to call the conference DevOpsDays. Videos of the talks will be online in the archives section soon, so I’ve decided to write down my thoughts about what went well and not so well – I am a fan of retrospectives.

What Went Well

The conference was in Ghent, a city in Belgium, and as expected had quite a heavy European contingent. It was not not exclusively so though, as Reductive Labs and Canonical both sent representatives over from the USA, and one of the speakers is an Aussie who just happened to be back packing around Europe at the time. One of my first great learnings though was that there are about 10 of us in the greater London area who are doing this kind of thing (2 of them ex-TWers) so we’ve started meeting regularly and exchanging ideas.

As there was such a large concentration of us around London, we’ve decided to try and subvert one of the London Geek Nights as a DevOps night to try and spread the word to more Devs. I’ll probably make more noise about it here when we’ve got dates confirmed.

The next great thing I loved was the format. It was split up into talks in the morning, with Open Spaces in the afternoon, over 2 days. This format actually worked very well as many of the open space sessions were inspired by the talks. Another interesting mix was that there was an even split between talks on tools and talks on practices.

I also discovered some great tools! Here’s a brief list of ones I liked the most:

• The Marionette Collective – using a message bus to administer 100′s of servers using pub/sub topics. You’ve got to watch the video to appreciate the full awesomeness.
• Cucumber-Nagios – writing tests to monitor your system using Cucumber, and then formatting the results as a Nagios plugin so you can easily wire it in to your monitoring system.
• Flapjack – another example of interesting use of a message bus. This time to parallelize the monitoring of 100′s of servers. I’m not sure I totally agree with the approach, but what really blew me away was they way they do graphing and reporting. It’s very impressive.
• collectd – collectd gathers statistics about the system it is running on and stores this information in RRD databases. One of the things I really like about this though is that there are plugins for it like Jcollectd which requires no code changes. Simply plug it into your classpath and it registers a JMX MBean that pumps data from your JVM to collectd (I think). I’ve not tried it in production yet though…

Overall the conference was very well received, and I’m looking forward to next years instalment. There’s a lot of talk about doing a North American version…

Not So Well

As we all know, the best bits about Away Days/Conferences are the talks in the hall between sessions. The problem with the program was that we barely had enough time for bio breaks between sessions, and everyone was quite burned out at the end of day two (not just because of the Belgian Beer). I think it’s a pretty easy thing to fix though and considering that it was the first conference of its type there’s bound to be some fine tuning required.

How many times have you come across a .jar or .dll file and wondered what version it is? Especially when using Open Source Software?

The most frightening one for me is when I’ve looked at a cluster of production servers and noticed that the .war file for the application deployed on it is a different size on one of the nodes. Which one was the correct one to deploy? Luckily this happened to me a long time ago, but I know that people out there are still having this problem today.

The solution is what I call “Self Identifying Software”. Every build of your software needs to have something that tells you what version it is and how to get back to the source code that created it. Having a build label or release number visible in your application is a good start, but it does not make your software Self Identifying. Product companies have been doing this for ever. The problem is that for that number to be useful (particularly when you’re trying to access the source code to reproduce and fix a bug) you then need to refer to a build system or release notes to find out where the source code came from (if you’re lucky). It often also does not apply to development builds. To be truly Self Identifying you need to make sure that every build (including builds developers create on their workstations) also includes enough information from the SCM system so that anyone who has access to the source code can go right back to the exact source code that produced that binary. For example, if you use Subversion as your SCM then this will be a URL and a revision number.

This is not exactly a new concept, it’s something I (and others) have been doing for a number of years now. The reason I’ve decided to write about it now though is that recently I was showing a new guy around one of the projects I’m working on at the moment, and when I showed him how to determine which version of the app was deployed he was delighted.

How do I get my software to identify itself?

There are generally two scenarios where I would want to get the identity of some software. The first is where I have a service or system running somewhere that I would like to know the version of, and the second is where I have a binary artefact and I’m trying to determine it’s provenance.

When querying a service, the easiest is to be able to hit a well known URL (or make a well known service call) that tells me what I need to know. What the exact URL or service call is is not important, as long as those who need it know how to invoke it. This is most useful then you don’t actually have access to the binaries that power the service.

However, if you do have access to the binaries then it’s usually easiest just to examine the meta data of the file to see where it came from. Most modern binary formats allow for this in some way. If you’re working with Java then all .jar, .war and .ear files allow you to put this into the META-INF/MANIFEST.MF file. Creative use of versioninfo resources will allow you to achieve similar results if you’re dealing with Windows. Note that we’re not talking about putting this information into the file name itself – file names change.

How do I use it?

Having Self Identifying Software goes beyond just being able to quickly track down the source code that built the binaries you’re dealing with. Here are a few other uses I’ve found:

• When used as part of a Continuous Deployment system you can verify that you’ve got the correct version of the binaries before you start deploying things, and you can then use the service call to verify that the correct version of the app is up and running at the end of the deployment processes.
• When used as part of a dynamic system dashboard you can quickly and easily see what’s installed where instead of relying on spreadsheets or documents that often get forgotten.

If you find any more, please let me know…

This is always a hard one to answer. In the last place I worked as a System Administrator we used Split-Horizon DNS as one of the ways to solve connecting to the correct host for a service. Our applications had hard coded hostnames for key services such as db.internal.domain, loghost.internal.domain and mailhost.internal.domain. Depending on which environment they ran in, when the did a DNS lookup they would get an answer that pointed them to the correct server providing that service for the environment they were physically running in. Combine this with tools like an LDAP Server in each of these environments containing the rest of the system configuration and life is easy. Or is it?

DNS and LDAP servers are relatively straight forward for experienced Sys Admins to understand and maintain, but they still can’t be described as easy to use, which is probably why so few people do it. This limits our options for fixing things down to two general categories – fix DNS and LDAP admin tools so they are easy to use, or find another way to solve the problem.

We chose the path of least resistance and went for the second option – creating ESCAPE. In the Unix tradition, it’s an app that takes one problem and solves it simply and well (at least in our opinion). It provides a RESTful interface for both the setting and getting of environment configuration. GET requests will retrieve configuration, whereas POST/PUT requests will create new entries or update them.

The URL scheme we’ve decided on is most easily demonstrated by constructing an example URL:

As the tool is now usable we’ve happily released version 0.1. Currently we’re busy putting a lot of work into making the management interface understandable and intuitive to use. The closest thing we currently have to a roadmap is our ToDo list.

Please join us on the discussion group with your feedback.