I’ve been a bit out of commission for the past month: starting a new job and moved a long way to be close to it. Before the move I had a bit of a backlog of work to write about and things have finally settled down enough for me to start working that off. This first post is about the work I’ve done to learn a bit about buildbot and how to use it to build some stuff I’ve been working on.

I’ve always been a bit of a “build junkie” and I’m not ashamed to admit that I really like to write Makefiles by hand. Learning buildbot was only partially motivated by masochism though. The other half was motivated by my need to build a few different projects including the meta-measured OE layer, the meta-selinux OE layer and OpenXT.

I’ve put this work up on github and it still needs some work before it’s portable to anything beyond my build setup. I hope to make this reproducible by others but before I do that I want to:

1. introduce these repos
2. cover some of the buildbot internals but only the bits required to make the build work

In doing this work I definitely fell down the rabbit hole a bit and started reading a pile of buildbot code but a lot of it was above and beyond what’s necessary to get this stuff building. I’m often guilty of wanting to understand everything about a piece of code before I start using it but that’s not always the right approach. I’ll do you all a solid and leave this stuff out to save you the hassle. This first post will cover the repo containing the buildbot config I’m using. My next post will cover some support code I’m using to trigger builds. I’ll focus on the bits necessary to build OpenXT initially and will include a quick write-up of my other side projects soon after.

twobit-buildbot

There are two repos that make up this work. The first is the actual buildbot config for both the build master and slave(s). It’s up on github here: twobit-buildbot. The second repo is a twisted python daemon that supports the build master and slave. I’m calling this last repo twobit-git-poller. This post will discuss this prior. My next post will cover the git poller.

Build Master

The buildbot config for both the master and the slave(s) I’m using are available on Github here: twobit-buildbot Buildbot configs are a bit hard to call “configs” with any seriousness since they’re proper python code. Still, it’s “config-like” in that there’s really no business logic here. It’s just a pile of classes that you filled with data and passed to the buildbot guts for execution.

If I go line by line through the config then this will quickly digress into a buildbot tutorial. There are plenty of those out there and if you’re interested in getting a deeper understanding then you should start with the buildbot docs. The steps that buildbot goes through to build a project are all wrapped up in a container called a BuildFactory. In my repo the BuildFactory for OpenXT is defined here. This class wraps an array of BuildStep objects that define a series of steps required to build the software.

The BuildStep objects are processed in the order they’re passed to the BuildFactory. I’ll run through these in order just to give you a feeling for the build progression:

1. The first thing in building OXT is to clone the build scripts. Buildbot knows where on the build slave to check this stuff out so all we do is provide some metadata to tell the buildbot git fetcher how we want the checkout to happen. OpenXT doesn’t yet support incremental builds so for now each new build requires a clean checkout.
2. Next we execute a shell command on the slave to set up the build config. To do this we use the ShellCommand class which is exactly what you’d expect: it executes shell commands. But with buildbot it’s always a question of where: on the master or on the slave? AFAIK for most build bot stuff you should assume that the action will happen on the slave unless otherwise noted. I’ve rigged the build master configs such that the script name isn’t used explicitly. Instead it’s a shell variable that the slave is expected to set here. I’ll describe this a bit more in the next section describing the build slave.
3. The next build steps are a series of shell commands that execute the series of build steps that make up the build of the OXT build. These are hard coded as the minimum necessary to create an installer iso. These are all invocations of the do_build.sh script. We could just as well just execute the do_build.sh script and have it get the build steps from the .config file but it’s nice to execute them one by one so that the buildbot UI shows them as unique steps. This also makes it easier to go through the output logs when things go wrong.
4. Once the main build steps are done we need to get the build output to a place where people can download it. So the next step runs a MasterShellCommand to create a directory on the build master. In the case of my build master this is on an NFS mount that gets served up by my webserver. That should tell you how we serve up the build output once it’s on the build master.
5. Now that we’ve got a place to stash the installer iso we need to move it from the build slave to the build master. OXT has some script functions in the build script to rsync build artifacts to various places. Buildbot has a build step to handle this for us so I’ve chosen to use that instead. Hopefully we’ll retire the bulk of the custom OXT scripts and use the tools available like the FileUpload class in buildbot.
6. Unfortunately I ran into a nasty buildbot limitation after getting the build upload to work. The version of buildbot packaged for Debian Wheezy runs the buildbot daemon with very paranoid umask (022). This was resolved 3 years ago but Wheezy doesn’t have the fix. I opted to hack around this with an extra build step instead of requiring people to install buildbot from source or apply a patch to the Debian packages. This hack is ugly but it just runs a scrip on the build master and fixes up the permissions on the newly uploaded file and directory housing it.

The BuildFactory describes how to build something but we still need something to do the building. This is the job of the build slave. The BuildlerConfig is an object that associates a build slave with a BuildFactory. When the build master determines that a build needs to be run the buildbot code will go through the available BuilderConfig objects, find the right config, and then walk the build slave through the right series of BuildSteps.

Build Slave

Naturally we assume that the build slave executing these steps is capable of building the project which in this case is OpenXT. That is to say you’ve followed the steps on the OpenXT wiki to set up a build machine and that you’ve tested that it works. Further, I’ve only tested this with the buildbot package from Debian Wheezy. The build master can be a simple Wheezy system, no need to jump through the hoops to make it an OXT build system since it won’t be building any code.

The config for a build slave is vastly more simple than the master. It only needs to be able to connect to the master to receive commands. The only really interesting bit of the build slave config is in a few of the environment variables I set.

Each of the OE images build my slave is building requires a sort of generic “setup” step where the slave sets some values in a config file. This is simple stuff like the location of a shared OE download directory or something specific to OpenXT like the location of the release signing keys. In each case there’s a script in a bin directory. For OpenXT that’s here.

The path to these scripts doesn’t need to be known by the build master. Instead the build master expects the slave to have a specific environment variable pointing to this script. The build master then tells the slave to execute whatever this variable points to. The build slave only has to set this variable properly in its config like this.

Wrap Up

That’s just a quick rundown of the buildbot bits I’ve strung together to build my pet projects and OpenXT. There’s one glaring gap in this: how buildbot knows when it needs to run builds. This is usually a pretty simple topic but with OpenXT there were a few things I had to work around. I’ll fill this in next time.

Till then I’ll be keeping the configs for my build master and slave on github and as up to date as possible. Feel free to reproduce this work in your own environment if you’re interested. I may or may not keep the build master exposed publicly in the future. For now it’s just intended to be an example and a convenient tool.