First OpenXT build

UPDATE; 2014-11-26 I’ve disabled comments on this post. All discussion about building OpenXT should be on the mailing list: https://groups.google.com/forum/#!forum/openxt
UPDATE; 2014-07-25 This page has been superseded by the documentation available on the OpenXT github wiki: https://github.com/OpenXT/openxt/wiki
UPDATE: 2014-06-20 added section about using bash instead of dash
UPDATE: 2014-06-19 fix git clone URI
UPDATE: 2014-06-18 add genisoimage package to list of required packages.
UPDATE: 2014-06-18 remove the section on mangling the manifest file now that I’ve upstreamed a patch.
UPDATE: 2014-06-18 to reflect default mirror being available now.
UPDATE: 2014-06-18 to clarify location of STEPS variable and the setupoe step.

With the transition of XT to OpenXT I’m realizing that the mark of most successful open source projects is good tools and great documentation. Right now we’re a bit short on both. The tools to build the core of OpenXT aren’t our own, they’re maintained by the upstream OpenEmbedded and Yocto communities. The documentation to build OpenXT on the other hand is our responsibility. This is a quick recap of my first build of the code that’s up on github with step-by-step instructions so that you can follow along at home.

The Existing Docs

The closest thing to build docs that we have on github are a README that was left over from a previous attempt to open source this code. That work was done under the name “XenClient Initiative” or XCI for short. This project was before my time on the project but my understanding is that it wasn’t particularly successful.

I guess you can call OpenXT our second go at the open source thing. The instructions in this file are way out of date. They need to be replaced and hopefully this write-up will be the first step in fixing this.

Build Machine

There’s a lot of technical debt that’s built up over the years in the OpenXT code base. The first and most obvious bit of technical debt is in our build system. We require that the build be done on a 32 bit Debian Squeeze system. The 64 bit architecture may work but it’s untested AFAIK.

We require Squeeze for a number of reasons the most obvious of which is our dependency on the GHC 6.12 compiler. Wheezy ships with version 7 and our toolstack hasn’t been updated to work with the new compiler yet. To the Haskell hackers out there: your help would be much appreciated. No doubt there are other dependencies and issues that would need to be worked around in an upgrade but we know this one to be a specific and prominent issue.

The requirement for a 32 bit build host is likely that only 32 bit build hosts have been tested. Any one out there who tries a 64 bit build or a build on Wheezy please report your results so we can get documentation together for the tested and supported build hosts.

Required Packages

The initial list of packages required to build OpenXT can be obtained from the OE wiki. The requirements are:

sed wget cvs subversion git-core coreutils unzip
texi2html texinfo docbook-utils gawk python-pysqlite2
diffstat help2man make gcc build-essential g++
desktop-file-utils chrpath

The list is pretty short as far as build requirements go because OE builds nearly all of the required tools as part of the build. This is a big part of what makes OE so great.

Additionally we require a few extra packages:

ghc guilt iasl quilt bin86 bcc libsdl1.2-dev liburi-perl genisoimage

Packages like guilt and quilt are used in our bitbake patch queue class in the expected way. ghc is the Hasekll compiler which is required to … build the Haskell compiler (much like OE requires gcc to build gcc). genisoimage is used by our build scripts to put the final installer ISO together.

The remaining dependencies: iasl, bin86, bcc, libsdl1.2-dev, and liburi-perl are another instance of technical debt. These packages should be built in OE as dependencies of other packages. Instead our recipes take a short cut and require they be installed on the build host. This seems like a smart shortcut but it’s a shortcut that ends in cross-compile misery. This may be what causes issues between 32 and 64 bit build hosts.

A good example of how to fix these issues already exists. If you’ve been following upstream Yocto development the Xen recipe contributed there gets this right depending on the dev86-native and iasl-native packages. OpenXT would benefit from pulling in the meta-virtualization layer and using this recipe (thanks Chris!)

Bash vs Bourne

Bitbake recipes contain a lot of shell functions and fragments. Per the these will be executed by the host systems /bin/sh. Unfortunately lots of build metadata (including the OpenXT build metadata) is rife with ‘bashisms’. Because of this, Linux distros that don’t link /bin/sh to /bin/bash will cause builds to fail.

The way to resolve this is laid out in the Ubuntu section of the “OE and Your Distro” docs as Debian and thus Ubuntu use the dash shell instead of bash by default. Switching dash for bash is pretty easy thankfully:

sudo dpkg-reconfigure dash

Executing the command above will result in a dialog screen asking you whether or not you want to use dash as your default shell. Select ‘No’ and your system will use bash instead.

I’ve gone ahead and filed a ticket to get ‘bashisms’ out of the first place I ran into them in OpenXT: https://github.com/OpenXT/xenclient-oe/issues/1. If you’ve got some time to kill it would be helpful if someone could track down more of our dirty laundry like this, or better yet, send in a pull request to sort some of this out.

Clone and Configure

If you’re following along you should now have a 32 bit Debian Squeeze build host with some additional packages installed. The next step is to clone the base OpenXT git repo:

git clone git://github.com/OpenXT/openxt.git

This will give you a directory named openxt that contains our build scripts. Change into this directory and we’ll take a look at the important files.

Firstly the file you’ll most often customize in here is the .config but you don’t have one yet. Copy the example-config file to .config so we can modify it for our environment:

cp example-config .config

The .config file is read by the script do_build.sh that … does the build. There are a hand full of variables in .config that are interesting for our first build. For now we’ll ignore the rest.

STEPS

We’ll start with STEPS. This one isn’t defined in the example-config but it’s initialized in the do_build.sh script. Since this script imports all variables from the config we can add it to the config manually to get the desired effect.

This variable defines the build steps that will be carried out when do_config.sh is run with no options. The default steps don’t all work yet so we’ll set this to the minimum that we can get away with for now:

STEPS="initramfs,stubinitramfs,dom0,uivm,ndvm,syncvm,installer,installer2,ship,copy"

I’ve left out the necessary setupoe step because I typically run this one manually and check that my variables get populated in the OE local.conf successfully. You don’t need to do it this way but it may help you get a better understanding of how the configuration works. After I go through the necessary variables we’ll go back to setupoe.

OPENXT_MIRROR

Due to some of the software used in OpenXT being a bit old the upstream mirrors of a few source tarballs are no longer available. Further we require a number of Intel binary ACM modules for TXT to function properly. Intel hides these zips behind a lawyer wall requiring users to accept license terms before they can be downloaded. That’s great CYA from their legal department but it prevents any automated build from downloading them in the build so we have to mirror them ourselves (which is permitted by their license).

When I did my first build the other day the URL from the example configuration file wouldn’t resolve for me. So I set up my own mirror:

OPENXT_MIRROR="http://www.twobit.us/mirrors/openxt"

This should be fixed so check the default mirror first. If it doesn’t work feel free to clone my mirror in your local environment but do me a favor and go easy on the bandwidth please.

UPDATE: The default mirror is fixed. You should use the default value (http://openxt.xci-test.com/mirror/) and not my mirror … well really you should set up your own mirror because there’s no way to tell how long the default mirror will stay up and it’s nice to keep redundant traffic to a minimum.

Signing Certs

The topic of signing releases is a huge one so for now we’ll just stick to the minimum work required to get our build signed once it’s done. There are several relevant config variables for this:

REPO_PROD_CACERT="/path/to/prod-cacert.pem"
REPO_DEV_CACERT="/path/to/dev-cacert.pem"
REPO_DEV_SIGNING_CERT="/path/to/dev-cacert.pem"
REPO_DEV_SIGNING_KEY="/path/to/dev-cakey.pem"

We require that each builder create their own keys for development and release builds. In this example I’m using self signed certs so it’s as simple as possible. Use the following commands to create your keys and your self signed certs:

openssl genrsa -out cakey.pem 2048
openssl req -new -x509 -key cakey.pem -out cacert.pem -days 1095

You’ll need two key/cert pairs, one for the automated signing of the build (a ‘dev’ key) and the certificate for a production signing key. All protection of the production signing key is the responsibility of whoever is signing the release. I’ll cover this in another post at another time. For now just make the keys and the certs and put the variables in the .config file.

OE_BUILD_CACHE_DL

For those of you who have used OE in a previous life you know how huge an OE build directory can get. Part of this is caused the the OE download cache which is the directory where OE caches downloaded source code. In the OE local.conf file this is specified by the DL_DIR variable. We use OE_BUILD_CACHE_DL instead.

Personally my build system has a RAID SSD set up to keep my builds as fast as possible but I can’t afford enough SSDs to support having my DL_DIR on my SSD storage. Typically I’ll use larger but slower storage (NFS RAID array) for my download mirror that I share between projects. Often times I’ll just link that slower storage mount directly into my build tree to keep everything in one place. Do whatever works best for you and remember this is completely optional. You can leave this out and the build will just use a directory in your build but remember this will make your build much larger:

OE_BUILD_CACHE_DL="/path/to/oe-download"

Even with the download cache on a separate volume an OpenXT build takes up a lot of disk space. This minimal build of just the core service VMs and no in-guest tools weighs in at 74G. My download cache is shared between all of my OE projects so I can’t say exactly how large the cache for a fresh OpenXT build will be. My combined cache is ~20G but I’d expect the files for OpenXT are a small portion of that.

Start Your Engines

That’s about all it should take. Next I run the do_build.sh script explicitly executing the setupoe step:

./do_build.sh -s setupoe

This step clones all of the git repos you need to build OpenXT (mostly OE meta-layers) and then processes your .config file to generate the OE local.conf. You can take a look at this file to see the variables and make sure they’re set to the right thing. This can be very helpful when debugging or if you want to manually change them for some reason.

After OE has been setup you can just run the do_build.sh script and then go drink coffee for the next 6 hours:

./do_build.sh | tee build.log

If all goes well at the end of this you’ll have an ISO with the OpenXT installer on it ready to go here:

./build-output/openxt-dev--master/iso/installer.iso

There’s no magic here. The comma separated list of steps we put into the STEPS variable can all be invoked one at a time the same way we ran setupoe. So if you want to build just the dom0 initramfs you can do so like this:

./do_build.sh -s initramfs | tee initramfs.log

Piping the build output to a file is always useful when things break (and they will break). My recommendation would be to build a few steps individually before you go and do a full build. I’ll probably do another post about the individual build steps and how all that works next.

Now go forth, build, install, and report bugs.