Story: Daniel's Continuous Integration System

Daniel Spiewak gave us this great story for the Atlassian Giveaway. He's earned a t-shirt. Have a good weekend, where-ever you are.

It was a dark and stormy night. No, actually it was a pleasant summer day, but daylight lacks a certain dramatic flare which is so necessary for a good story, especially a story about build systems.

I was working as the semi-lead developer for a mid-sized project run out of London, UK. My job was primarily to work on the Java clone of the Cocoa client application. Through a very clever and dangerous bit of hackery, the Cocoa and Java clients shared a single, Java-based backend which communicated with the server via xml-rpc. Because of the project's architecture, there were a number of inter-dependent sub-projects. As I was working on a clone of the Cocoa client, it was often necessary for me to build a new copy of the client after each new change. However, this was, in and of itself, a non-trivial undertaking. Once you added the building of the other sub-projects both individually and as dependencies, and my days started to look more and more like the "dark and stormy" variety.

Now, each project (with the exception of the Cocoa frontend) had an Ant build script which I had carefully crafted. These build scripts would invoke each other as need be, meaning that I could build a copy of my Java clone by simply invoking its build and allowing Ant to handle the rest. This solved a lot of my dependency headaches, but building every single project was still a tedious undertaking. Thus, I build another layer of abstraction above Ant, consisting primarily of Ruby scripts hacked together on top of Ant. The idea was that I could invoke my master build script, passing a list of project descriptors, and this build script would determine the optimal way to build each project and its dependencies. I was even able to rig this script with cron so that it automatically built a new version of each sub-project as necessary.

Unfortunately, this build script worked a little too well. My boss got wind of it and decided that it should be put onto the main development servers as a sort of continuous integration solution. This sounded like a good idea at the time, but it ultimately led to far more trouble than it was worth. I got sucked into the position of permanent build system maintainer; and, given the hacky nature of the system's architecture, it ended up being quite the position. As more sub-projects were added and more flexibility was needed, I actually had to rewrite the entire system from scratch more than once. Looking back, I'm actually astonished by the sheer number of hours I spent cursing those scripts into behaving.

I was probably on my third or fourth rewrite before I realized the idiocy of what I was doing. I had literally created a full continuous integration build tool from scratch (complete with web admin system and XML-RPC API) without ever considering the alternatives. It only took me a few minutes of research to realize that Hudson, Cruise Control, Bamboo, and really any CI system would solve exactly the same problems without any need for hacky scripts or unnecessary effort. It took my boss a little while longer to come around, but eventually he too saw the wisdom of relying on a pre-existing solution rather than rolling our own convoluted hack job.

The really amazing part of this story is how I didn't even see what I was doing until very late in the process. It started out as just an innocent collection of scripts to aid my own development process. Each step I took toward hacky maintenance hell was so gradual, so subtle in form that I completely failed to see where I was headed until I was already there. An while my build system didn't actually require a defunct Pentium series processor to run, it does certainly certainly qualify as a bizarre polyglot, home-grown build system which should never have been allowed to fester.

DevOps New Zealand