Tag: OpenEmbedded

meta-measured updates

~ flihp ~ Leave a comment

I’ve been grinding down updates to meta-measured slowly but surely over these past few days. This was started a while back but slowed to a stop when I got distracted by work on meta-selinux. Last week my work to upgrade the SELinux toolstack was merged upstream so I’m re-focusing on getting meta-measured back in shape. This is a quick run down on the changes.

General Maintenance and Release Tracking

I haven’t been chasing upstream much so most of this work has just been maintenance. The kernel in OE has moved forward with 4.1 the new stable in master. The read-only root file system class was broken when I tried a fresh master build the other day. When jumping in to start a refresh, finding core features broken can be a bit discouraging. Instead of taking on this fix immediately I figured it would be best to create a branch to track the latest release (fido) and get that working first. I’ve always avoided tracking OE releases because I don’t want to give the impression that meta-measured is in any way “stable”, as if having my master branch broken on a regular basis wasn’t a good enough indicator.

Seriously though, I kinda dropped the “chasing master” ball. Having a branch that follows the latest release is a good way to make getting everything up to date and it’s a good insurance policy against future breakage in master. So win / win on that. Fido still has a functional readonly-rootfs class so, for the 2 other people out there who have kicked the tires on meta-measured, fall back to a release branch. My goal is to keep at least one release branch in working order.

There are two changes between fido and the current master that have an impact on meta-measured. The first is the upgrade to GCC5. For the trousers and tpm-tools packages this requires a patch that I’ve gratefully borrowed from the hardworking people over at the Debian project. The other is meta-intel nuking the meta-nuc layer. This is a pretty minor change and I’m now using the measured-intel-corei7-64 that extends the intel-corei7-64 machine from meta-intel on the master branch. Fido still has the NUC machine so I’ll stick to that for Fido.

systemd

The first external submission to meta-measured came in a few months back. This was to add systemd support to the trousers package. I had done something similar to this as part of some work to get meta-measured working with the Edison but it was in the BSP layer. It really shouldn’t have been in the BSP layer to begin and my implementation was a hack regardless. The patch I received that adds this feature directly to meta-measured was well done so I’ve added it directly and removed my hack from the BSP.

The only bit I didn’t like from the patch was a utility script that loaded every TPM module under /lib/modules/$(uname -r). This seems a bit heavy handed given that modern Linux kernels autoload these modules when the device is detected. Pretty minor change though and this is now in meta-measured proper.

IMA and TPM2

I started playing around with configuring IMA a while back as a way to exercise the TPM2 that’s baked into the Minnowboard Max as part of PTT. I don’t have any specific plans to do much with IMA just yet but with the xattrs patches I pushed into meta-selinux I’m hoping to be able to support IMA appraisal. It would be a great use case and demonstration to show the utility of my xattr work with an OE image booting with IMA appraisal enforced from firstboot.

UEFI Breakage

I’ve been saving the bad news for last unfortunately. Somewhere in the changes that have been going on with OE, my bringing meta-measured up to date and flashing the firmware on my IVB NUC, measured-image-bootimg broke. I’m pretty sure this is a problem with the latest firmware for the NUC but that’s a guess. Grub2 comes up OK but tboot doesn’t seem to take over.

In prevous debugging I learned that tboot won’t log to VGA since UEFI doesn’t setup a text console for use by whatever executes after ExitBootServices() is called. It’s frame buffer only in UEFI world I guess. But tboot would still log to a serial device through direct I/O. Not this time though. Once Grub2 kicks off tboot my NUC is unresponsive and I get no additional output. Booting straight to the kernel from grub works fine but going the tboot route through multiboot2 isn’t working.

I’m pretty much out of debugging ideas here. My test system is getting a bit old and it’s the only UEFI system I have for use here so I’m writing this off as a loss for the time being. If anyone out there is testing this on a UEFI system I’d love to hear about whatever results you have to report.

Next steps

I’ve added the contributed systemd support to trousers but I haven’t enabled or tested systemd support for my reference images. This is next on my list of things to do. I’ve got trousers working on a systemd driven image previously in some work to wire a TPM up to an Edison system (more on this in the future hopefully) but I was working on on the reference Edison images. Stay tuned and hopefully I’ll have proper systemd support in the near future.

xattrs in OE rootfs part 2

~ flihp ~ Leave a comment

This is an update on some work from a previous post. My xattr patches sat on the yocto mailing list waiting for meta-selinux maintainers to review them for about a month. I got a bit testy and even commented to another contributor off list about the possible need to fork given the lack of response from maintainers. Turns out that this contributor was using his gmail account but actually works for the same company as the maintainer. So about 24 hours later the maintainers magically reappeared and merged my patches!

I really have no interest in forking and maintaining meta-selinux so this was a win. Hopefully I didn’t look like too much of a jerk in the process … ¯_(ツ)_/¯ After a quick stint in a feature branch my xattr code was merged into meta-selinux master. This is only a step in the process though. These patches really belong in the openembedded-core repository. They can likely be moved with minimal effort on the technical side. Getting traction for these sorts of fringe features upstream however takes a bunch of time though, as demonstrated by the 2 months it took to get this into meta-selinux.

As a strategy I should probably prove out an additional use case beyond SELinux file labels. I did some work on meta-measured a while back to add basic IMA support. I really want to expand this to support IMA appraisal. I’m not a huge fan of local enforcement policies but this use case is a great example of build-time xattr usage that would demonstrate the utility of my xattr patches. This is all food for thought though at this point.

OXT and Paying Down Technical Debt

~ flihp ~ Leave a comment

The few people out there who have slogged through the process of building OXT know how painful it is / can be. One of the most significant contributors to the difficulty here is the amount of technical debt that has built up in the build system over the years. We always “got it to work” when doing build stuff, but with the rate of change in the OE project / community we never kept pace with “doing it right”.

The reasons for this are many and every developer out there is familiar with the “ship it” pressures that tip the scales from “do it right” to “just make it work”. I won’t get into that here because it’s just the way things are now. What’s interesting and useful IMHO is coming up with a way to tip the scales back now that the pressures / incentives are different. So this post will cover some of my work to pay down some of the technical debt that I think is particularly important.

File Systems and Xattrs

One of the unique opportunities afforded us by the OE build system is the ability to produce complete file systems. When we do an install of OXT we don’t have to muck around in the file systems of any of the service VMs (including dom0) because they’re all ready to go straight from the build system. This means we can boot them read only from first boot. With a little bit of work, we could even do a measured launch from first boot complete with predictable measurements.

One of the hurdles we had to overcome was the need for xattr support in the file system creation code. Refusing to have a bunch of “first boot” code to put band aids on the shortcomings of our build system means that we had to come up with a way to create xattrs in the build and have them preserved through the creation of the service VM file systems. This is a problem that the usptream meta-selinux OE layer suffers from currently as anyone who’s built the core-image-selinux image will tell you. They have first boot code that detects the unlabeled file system, relabels it accordingly and then reboots. Only after this first reboot will the system come up with SELinux functioning properly.

Upstreaming xattrs into e2fsprogs

Having some common ground with an upstream like this seems like a good first step. Nothing like a shared problem / need to motivate work. For the OXT community getting this upstream will get us one step closer to being able to build service VMs for OXT (including our own) that don’t depend on a pile of old build metadata and an ancient version of bitbake.

The bits that need to go upstream are actually pretty small. We had to hack up a patch to Pseudo to get that to support xattrs but upstream Pseudo has already implemented this functionality and they did it right. When I saw that Pseudo had already implemented this I was relieved as it probably cut the amount of effort required in half. This left getting the actual disk image produced by the build to support xattrs.

gene3fs was the OXT tool that we used to get the build to spit out ext3 file systems. This is a departure from upstream where they have patches added to the e2fsprogs mke2fs utility. This work wasn’t mine but gene3fs looks like a fork of the unmaintained gene2fs project adding support for the ext3 file system, xattrs and VHDs. Even if this approach is technically superior to the usptream method it’s a moot point. Upstreaming is about longevity.

So ignoring the VHD stuff for now I’ve implemented a set of patches for OE that add xattr support to the build. With these patches in place we can apply SELinux labels to a root file system and boot the resulting image into enforcing mode without having to do first boot fixups. Stage one complete!

Getting patches merged

The hard part about sending stuff upstream is motivating maintainers to merge the patches. meta-selinux is pretty sparsely maintained unfortunately and 3 weeks after having sent my patches to the Yocto mailing list I haven’t had anyone test or even respond to the post. I’m not sure which is worst: being ignored or having someone tear apart the patches because they suck.

Trying to keep from being too persistent / obnoxious I’ve left these patches alone for a bit. Thankfully the e2fsprogs package is one of those critical pieces of the build system and so the maintainers seem to favor stability over frequent updates. This means the likelihood of having to rebase my patches is pretty low. If I don’t get some attention from the meta-selinux maintainers soon I’ll likely rebase them into the openembedded-core layer and send them there. I think the need for this functionality is less understood there but they’re much more active and likely to provide feedback (even if it’s “no thanks”).

Tracking work

We haven’t had a lot of upstreaming work done for OXT yet so there’s not really a process in place for tracking such efforts. As such I’ve created a place on the confluence wiki where I’ll track this stuff. The SELinux / xattr specific stuff is here: https://openxt.atlassian.net/wiki/display/~flihp/SELinux+file+system+labeling+in+build
I’ve set up a page describing the general process I’m following as well and that’s here: https://openxt.atlassian.net/wiki/display/~flihp/Upstreaming

If anyone else is working on sending functionality upstream regardless of where feel free to work off these pages. Once we’ve got a few examples to work from we may decide to formalize the process but for now I’m just trying to pay down some technical debt and make some progress both for OXT and one of the usptreams we share common goals (and hopefully code) with.

Attaching a TPM on the LPC

~ flihp ~ 1 Comment

Just for the funs I recently revived some older work with my PCEngines alix3d2 where I built an OE meta layer with a simple machine and kernel config to build images: meta-alix.

IMG_20150213_151432

IMG_20150216_183119

IMG_20150216_183320

TPMs for all the boards!

I’ve got a soft spot for the older PCEngines WRAP board since it was the first platform I experimented on while building a home router / access point years ago. So meta-alix was fun work but nothing too crazy. While playing around with this I noticed that the alix3d2 has an exposed 20 pin header labeled ‘LPC’. Now that is interesting because I’ve got a few Asus branded Infineon TPMs laying about and they’ve got LPC connectors on them. A home wireless router with a TPM on it? Now that could be interesting.

Attaching an TPM designed to attach to a board on a 20 pin LPC connector should be pretty easy right? That’s what I thought too. But here we are 2 weeks later and I’m just now getting to write this up and I can’t say this work was 100% successful. But before I go too deep into the trials and tribulations let’s start with a bill of materials.

Bill of materials

To start out you’ll need a TPM and one designed to attach to your system on the LPC bus. TPMs are a PITA to buy really. There are 3 main companies that manufacture them but you can’t buy them direct. Thankfully there are some motherboard manufacturers out there that support the TPM via a “daughter-card” and from my experience this is mostly the high end manufacturers like Asus and Supermicro. I had 2 Asus TPMs laying around so this seemed like a good opportunity to put them to use. On Amazon these TPMs go for about $15 but when I bought mine almost a year ago they were less than half that.

The system that started out trying to attach this thing to is an alix3d2. I also picked up one of the newer PCEngines APU but *spoiler alert* only after I had serious problems getting the alix to work.

You’ll also need a soldering iron and the usual soldering / prototyping gear on hand (lights, wire, solder, magnifying glass etc). That’s right I said soldering. It’s been a while for me too. Like 10 years. Don’t worry there isn’t much too this and it was really fun.

Prototyping

As you’ve likely guessed by now, just because a system has an LPC connector doesn’t mean this thing is plug and play. The Asus TPM daughter card has pin 4 blocked / keyed and the ALIX doesn’t so that’s our first hint. The real data is in the respective pin diagrams. Finding these isn’t as easy as I’d hoped so I had to do some digging.

The docs for the ALIX systems are all on the PCEngines website so that part’s easy. The Asus TPM doesn’t seem to have any docs though. If you take the time to dig into the boards that support them though you’ll find the manuals for these boards have the pin assignment documented. I pulled down the manual for the P9D-WS and used this as a reference. Page 2-29 has what we’re looking for.

Pin Layouts

With the pin layouts in hand we can see clearly that plugging the TPM daughter card directly into the board isn’t gonna happen. I’ll reproduce the layouts here so we can view them side by side:

Asus TPM PCEngines LPC
pin signal signal
1 PCICLK PCICLK</td
2 GND GND
3 FRAME LAD0
4 BLOCKED GND
5 PCIRST# LAD1
6 NC GND
7 LAD3 LAD2
8 LAD2 GND
9 +3V LAD3
10 LAD1 GND
11 LAD0 LFRAME#
12 GND GND
13 NC PCIRST#
14 NC CLK48A
15 +3VSB ISP
16 SERIRQ Vcc (+5V)
17 GND GND
18 CLKRUN V3
19 PWRDWN SERIRQ
20 NC LDRQ#

There’s basically no overlap in the pin layouts here except for a few ground connections. This blew my mind at first but after searching through the Intel Low Pin Count Interface Specification it turns out that this bus was intended for use on-board only and so there’s no pin layout specified for external connectors. First mystery solved. Now let’s figure out how we’re gonna wire this thing up.

To the breadboard!

This isn’t going to be as easy as “plug and play” but it’s not far off. We just need to connect the right pins. With the pin map above and a little help from the spec (to get the minimum required connections) we can pull out our breadboard and prototype this thing.

If you’re like me you’ll have to go out and buy materials as you need them. Luckily I live minutes away from HSC Electronic Supply which is an amazing surplus electronic shop. After an hour or 3 poking around the piles of old electronic gear I managed to scrounge up a 20 pin ribbon cable with a connector that looked like it might fit on my breadboard. With a 20 pin DIP ribbon cable connector I had what I needed to connect the alix to the breadboard.

Next was to get the TPM daughter card wired up to the breadboard. This was harder than I expected. I couldn’t easily find a connector that would suit this purpose that didn’t require waiting for shipping. So I soldered some wires up to breakaway headers and rigged up a horrible TPM-to-breadboard connector. Then we just hook up the two using the following mapping:

TPM ALIX Signal
1 1 PCICLK / LCLK: 33MHz clock
3 11 LFRAME#: Transaction signal
5 13 LRESET#: Bus reset. AKA PCIRST#
7 9 LAD3: Data lane
8 7 LAD2: Data lane
9 & 15 18 3 Volts DC
10 5 LAD1: Data lane
11 3 LAD0: Data lane</td
16 19 SERIRQ: Serialized interrupt signal

After some fiddling (kicking, screaming and burning myself with a soldering iron) this is what it looked like:

IMG_20150213_093921

IMG_20150212_175548
Now it SHOULD have worked. These are the right connections. But on the alix3d2 I got no love. I didn’t actually get this set-up to work till my apu1d showed up in the mail 3 days later. For whatever reason the external LPC on the alix3d2 just doesn’t work as advertised. Without an oscilloscope I can’t debug much beyond whether the voltage and ground pins are OK (and they are) so for now that will remain a mystery. So the alix3d2 is out and the apu1d is in.

Anyways we can do better than this bootleg breadboard setup. Let’s see about cleaning it up.

IMG_20150213_093453
IMG_20150213_093638

Clean it up

The wiring above was super flaky and that shouldn’t be a surprise. I didn’t get the length of each wire exact and the pins slipped around a bit in the plastic. I ordered some IDC breakout helpers from Adafruit but they were garbage. They plug into the breadboard fine but the pins aren’t long enough and they just pop back out immediately.

So again I hacked up another connector out of DIP/DIL headers and some breakaway headers spaced to span the gap in the breadboard. This is generally a bad idea since the solder is what’s holding the whole thing together but it worked out pretty OK:

IMG_20150213_151417

IMG_20150216_183250

IMG_20150214_134235

Packaging for the APU enclosure

After convincing ourselves that the wiring above is right on the breadboard we need to clean this up so that it fits in the enclosure with the APU. There’s not a lot of space in the PCEngines recommended case1d2 but there’s enough if we’re sufficiently inventive. And by “inventive” I mean “you’ve got a hacksaw”.

Start out by removing the female header from the TPM and trim back the connector pins. If we flip this header on it’s side we can use it to mount the TPM once we reconnect it. This would require either unblocking pin 4 on the connector or cutting pin 4 off of the APU board. Since pin 4 on the APU is ground anyways this shouldn’t be a problem.

I used a 20 pin DIP to ribbon cable connector for my setup. I sanded down the daughter board to expose the copper on the base which happens to be ground and connected this with the even pins on the connector up through 12. This proved to be a pretty solid base as it holds the daughter board nice and tight to the connector.

Then we just cut wires and solder pins per the table above. The wire I had on hand was 28 gauge which was a bit too big and the soldering job is straight up ugly in spots but it’s the first bit of soldering I’ve done in 10 years so that’s good enough for me. I’ve got another TPM on hand so I’ll have another go now that I’ve had some practice.

Testing

I used both a Debian install with the tpm-tools package to test this as well as the core-image-tpm from meta-measured. I’d recommend sticking with Debian unless you feel like falling down the rabbit hole of an OE build. The important thing to keep in mind is that the APU BIOS doesn’t support the TPM so it won’t do the necessary setup for us.

The BIOS is supposed to do a number of things to set things up so that the OS can use the TPM. This includes running the TPM self test, enabling it and setting up ACPI entries to make it easy for the OS to talk to it. With the stock BIOS on the APU we won’t get any of this. Thankfully the number of platforms that have implemented TPM support wrong in the BIOS over the years is quite high so the Linux TPM TIS driver can do all of this for us if we give it the right parameters:

root@apu:~# modprobe tpm_tis force=1
[   74.027383] tpm_tis tpm_tis: 1.2 TPM (device-id 0xB, rev-id 16)
[   74.063388] tpm_tis tpm_tis: Issuing TPM_STARTUP
[   74.260392] tpm_tis tpm_tis: TPM is disabled/deactivated (0x7)
[   74.308465] genirq: Flags mismatch irq 4. 00000080 (tpm0) vs. 00000000 (serial)
[   74.315956] tpm_tis tpm_tis: Unable to request irq: 4 for probe
[   74.436459] genirq: Flags mismatch irq 8. 00000080 (tpm0) vs. 00000000 (rtc0)
[   74.443753] tpm_tis tpm_tis: Unable to request irq: 8 for probe

The modinfo command will tell you all of the gory details about what these parameters do if you’re interested. The short version is that force=1 causes the driver to ignore ACPI and probe for the TPM device. You can also add the interrupts=0 argument to disables interrupts which will get rid of all of the genirq errors. After this you should see /dev/tpm0 appear magically. You can then start tcsd and get some version info out of the TPM:

root@apu:~# tpm_version 
  TPM 1.2 Version Info:
  Chip Version:        1.2.3.19
  Spec Level:          2
  Errata Revision:     2
  TPM Vendor ID:       IFX
  Vendor Specific data: 0313000b 00
  TPM Version:         01010000
  Manufacturer Info:   49465800

You won’t be able to do much more than this though since the BIOS hasn’t enabled the TPM for us. We’ll get past this in my next post.

tboot 1.8.0 and UEFI

~ flihp ~ Leave a comment

Version 1.8.0 of tboot was released a while back. This is a pretty big deal as the EFI support has been a long time coming. Anyone wanting to use tboot on a modern piece of hardware using EFI has been out of luck till now.

For the past week or so I’ve been slowly figuring out how to build an OE image with grub-efi, building the new version of tboot and then debugging an upgrade in meta-measured. My idea of a good time for sure.

As always the debugging was the hardest part, building the software was easy. For the most part tboot EFI “just worked” … after I figured out all the problems with kernel version and grub configuration. Hard parts were

  • realizing the Linux kernel image had to be the latest 3.14 version
  • debugging new kernel version
  • configuring grub
  • which modules needed to be built into grub

If you want the details you can see the full history on the meta-meausred github. The highlights are pretty simple:

multiboot2 in oe-core grub-efi

The grub-efi recipe in oe-core is a bit rigid. I’ve pushed a patch upstream that allows another layer (like meta-measured) to modify which grub modules are built into the grub EFI executable. It’s a tiny change but it makes all of the difference:

http://lists.openembedded.org/pipermail/openembedded-core/2014-April/091768.html

This lets us add modules to the grub EFI executable. I also had to cobble together a working grub multiboot2 configuration.

linux-yocto v3.14

Pairing this with the older 3.10 Yocto Linux kernel image will allow you to get through grub and tboot but the kernel will panic very early in the boot process. The newer 3.14 doesn’t suffer from this limitation.

The measured reference image in meta-measured used aufs to keep from having to mount the rootfs read/write. This is to keep the rootfs hash from changing across boots. I wrote the whole thing up a while back: http://twobit.us/blog/2013/01/meta-measured/. Anyways aufs doesn’t work in 3.14 so I took the extra few minutes to migrate the image to use the read-only-rootfs IMAGE_FEATURE. This is a good thing regardless, aufs was being used as a shortcut. I hadn’t had the drive to fix this till it broke. Problem solved.

rough edges

I still haven’t figured out all of the details in grub and it’s configuration. The current configuration in meta-measured is sufficient to boot but something gets screwed up in setting up VGA output for tboot and the early kernel output. Currently grub displays an error message indicating that tboot won’t get a console and no VGA output will be shown till the kernel loads the DRM driver. Output is still available on the serial console so if you’ve got a reasonable test setup you can get all the data you need for debugging.

No lies, I’m a bit afraid of grub, guess I’ll have to get over it. The measured-image-bootimg has a menuentry for tboot and a normal linux boot. Booting the kernel using the linux and initrd grub commands provide normal VGA output but the multiboot2 config required by tboot does not. I take this to mean that grub is capable of doing all of the necessary VGA stuff but that it can’t pass this data through to tboot via multiboot2. More to come on this soon hopefully.

Till then, if you build this stuff and have feedback leave it here.

OE image package part 2

~ flihp ~ Leave a comment

I spent the weekend in bed with a cold … and with a laptop hacking on ways to get OE rootfs images packaged and installed in other OE rootfs images. My solution isn’t great but it’s functional and it doesn’t require that every image be built in series. But before I get too far let’s go over what I set out to achieve and get a rough set of requirements:

  • I’m building a rootfs that itself contains other rootfs’. The inner rootfs’ will be VMs. The outer rootfs will be the host with the hypervisor (Xen dom0).
  • Build speeds are important so I’d like to share as much of the build infrastructure between VMs as possible.
  • Running builds in parallel is a good thing. Running all builds as serial operations is a non-starter. Bonus points for being able to distribute them across multiple hosts.
  • Having to implement a pile of shell script outside of bitbake to make this work means you’re doing it wrong. The script that automates this build should be doing little more than calling bitbake.

First things first: my solution isn’t perfect. It does work pretty well though and achieves much of the above. Below is a quick visual of what I intend for the end product to support:

rootfs image relationships

On the left is the simple case I’m working to support currently. The boxes represent the root file systems (rootfs) that bitbake is churning out. The lines pointing from one rootfs to another represent one rootfs being packaged in another. dom0 here would be a live image and it would boot the NDVM automatically. Naturally the NDVM rootfs must be contained within dom0 for this to work. The right hand side is an eventual goal.

To support what most people think of as a ‘distro’ you need an installer to lay things down on a physical disk and if users expect to be able to run arbitrary workloads / VMs then they’ll want the whole disk available. In this scenario the installer image rootfs will have the image packages for the VMs installed in it (including dom0!). The installer when do it’s thing laying dom0 down in a partition but it can also drop the supporting VMs images into another partition. After installation, dom0 is booted from physical media and it will be able to boot these supporting VMs.

Notice the two level hierarchical relationship between the rootfs images in the diagram. The rootfs’ on the lower part of the diagram are completely independent and thus can be built in parallel. This will make them easily distributed across multiple build systems. Now on to some of the methods I tried to realize this goal and eventually one that worked!

Changing DISTRO in a build tree

The first thing I played around with was rewriting my local.conf between image builds to change the DISTRO. I use a different DISTRO configs to make package customizations that differentiate my images. This would allow me to add a simple IMAGE_POSTPROCESS_COMMAND to copy service VM rootfs images into the outer image (again, dom0 or an installer).

I had hoped I’d be able to pull this off and just have bitbake magically pick up the differences so I could build multiple images in the same build tree. This would make my image builds serial but possibly very efficient. This caused several failures in my tests however so I decided it would be best to keep separate builds for my individual images. I should probably find the right mailing lists to help track down the root cause of this but I expect this is well outside of the ‘supported’ bitbake / OE use cases.

Copying data across build trees

As a fall-back I came up with a hack in which I copy the needed build artifacts (rootfs & kernel image) to a common location as a post processing step in the image recipe. I’ve encapsulated this in a bbclass in anticipation of using the same pattern for other VM images. I’ve called this class integral-image-export.bbclass: 

inherit core-image

do_export() {
    manifest_install() {
        if [ ! -z "$1" ]; then
            install -m 0644 "$1" "$4"
            printf "%s *%sn" "$(sha256sum --binary $1 | awk '{ print $1 }')" "$2" >> $3
        fi
    }

    # only do export if export dir is defined
    if [ ! -z "${INTEGRAL_EXPORT_DIR}" ]; then
        ROOT="${INTEGRAL_EXPORT_DIR}/${PN}-$(date --utc +%Y-%m-%dT%H:%M:%S.%NZ)"
        FS_FILE="${IMAGE_BASENAME}-${MACHINE}.ext3"
        KERN_FILE="${KERNEL_IMAGETYPE}-${MACHINE}.bin"
        KERN_PATH="${DEPLOY_DIR_IMAGE}/${KERN_FILE}"
        MANIFEST="${ROOT}/manifest"
        mkdir -p ${ROOT}
        manifest_install "${KERN_PATH}" "${KERN_FILE}" "${MANIFEST}" "${ROOT}"
        manifest_install "${ROOTFS}" "${FS_FILE}" "${MANIFEST}" "${ROOT}"
    fi
}

addtask export before do_build after do_rootfs

It lives here https://github.com/flihp/meta-integral/blob/master/classes/integral-image-export.bbclass. So by having my NDVM image inherit this class, and properly defining the INTEGRAL_EXPORT_DIR in my builds local.conf, the NDVM image recipe will copy these build artifacts out of the build tree.

Notice that the destination directory has an overly precise time stamp as part of its name. This is an attempt to create unique identifiers for images without having to track incrementing build numbers. Also worth noting is the manifest_install function. Basically this generates a file in the same format as the sha*sum utilities with the intent of those programs being able to verify the manifest.

Eventually I think it would be useful for a manifest to contain data about the meta layers that went into building the image and the hashes of the git commit checked out at the time of the build. This later bit will be useful if a build ever has to be recreated. Not something that’s necessary yet however.

Consuming exported images

After exporting these build artifacts we have to cope with other images that want to consume them. My main complaint about using a build script outside of my built tree to place images within one another is that I’d have to re-size existing file systems. Bitbake already builds file systems so resizing them from an external script seemed very ugly. Further changes to the images built by bitbake (ext3/iso/hddimg etc) would have to be coordinated with said external script. Very ugly indeed.

The most logical solution was to create a new recipe as a way to package the existing build artifacts into a package that can be consumed by an image. By ‘package’ I mean your typical ipk or rpm. This allows bitbake to continue to do all of the heavy lifting in image building for us. Assuming the relationships between images shown above, it allows the outer image to include the image package using the standard IMAGE_INSTALL mechanism. That feels borderline elegant compared to rewriting the generated file systems IMHO.

So from the last section we have builds that are pumping out build artifacts and for the case of our example we’ll say they land in /mnt/integral/image-$stamp where $stamp is a unique time stamp. On the other hand we need to create a recipe that consumes the artifacts (I’ll call it an ‘image package recipe’ from here out) in these directories. Typically in a bitbake recipe you’ll provide a URI to your source code in the SRC_URI variable and define the files that go into the image using FILES_${PN}. These are generally defined statically in the recipe. Our case is weird in that we want the image package recipe to grab the latest image exported by some other build. So we must dynamically generate these variables.

Though I’ve never seen these variables generated dynamically (aside from using the PN and PV variables in URIs) but it’s surprisingly easy. bitbake supports anonymous python functions that get run when the recipe is parsed. This happens before any tasks are executed so setting SRC_URI and PV in this function works quite well. The method for determining the latest images that our build has exported is a simple directory listing and sorting operation:

python() {
    import glob, os, subprocess

    # check for valid export dir
    INTEGRAL_EXPORT_DIR = d.getVar ('INTEGRAL_EXPORT_DIR', True)
    if INTEGRAL_EXPORT_DIR is None:
        bb.note ('INTEGRAL_EXPORT_DIR is empty')
        return 0
    if not os.path.isdir (INTEGRAL_EXPORT_DIR):
        bb.fatal ('INTEGRAL_EXPORT_DIR is set, but not a directory: {0}'.format (INTEGRAL_EXPORT_DIR))
        return 1

    PN = d.getVar ('PN', True)
    LIBDIR = d.getVar ('libdir', True)
    XENDIR = d.getVar ('XENDIR', True)
    VMNAME = d.getVar ('VMNAME', True)

    # find latest ndvm and verify hashes
    IMG_NAME = PN[:PN.rfind ('-')]
    DIR_LIST = glob.glob ('{0}/{1}*'.format (INTEGRAL_EXPORT_DIR, IMG_NAME))
    DIR_LIST.sort (reverse=True)
    DIR_SAVE = os.getcwd ()
    os.chdir (DIR_LIST [0])
    try:
        DEV_NULL = open ('/dev/null', 'w')
        subprocess.check_call ('sha256sum -c manifest', stdout=DEV_NULL, shell=True)
    except subprocess.CalledProcessError:
        return 1
    finally:
        DEV_NULL.close ()
        os.chdir (DIR_SAVE)

    # build up SRC_URI and FILES_${PN} from latest NDVM image
    d.appendVar ('SRC_URI', 'file://{0}/*'. format (DIR_LIST [0]))
    d.appendVar ('FILES_{0}'.format (PN), ' {0}/{1}/{2}/*'.format (LIBDIR, XENDIR, VMNAME))

    # set up ${S}
    WORKDIR = d.getVar ('WORKDIR', True)
    d.setVar ('S', '{0}/{1}'.format (WORKDIR, DIR_LIST [0]))
    
    return 0
}

If you’re interested in the full recipe for this image package you can find it here: https://github.com/flihp/meta-integral/blob/master/recipes-integral/images/integral-image-ndvm-pkg.bb

The ‘manifest’ described above is also verified and processed. Using the file format of the sha256sum utility is a cheap approximation of the OE SRC_URI[sha256sum] metadata. This is a pretty naive approach to finding the “right” image to package as it doesn’t give the outer image much say over which inner image to pull in: It just grabs the latest. Some mechanism for the consuming image to specify which image it consumes would be useful.

So that’s about it. I’m pretty pleased with the outcome but time will tell how useful this approach is. Hopefully I’ll get a chance to see if it scales well in the future. Throw something in the comments if you get a chance to play around with this or have thoughts on the topic.

OE image package

~ flihp ~ Leave a comment

Here’s a fun problem that I don’t yet have a solution to: I want to build a single image with OE. This image will be my dom0. I want to include other images in this image. That is to say I want to package service VMs as part of / in my dom0.

All of the research I’ve done up till now (all 30 minutes of it) points to this having never been done before. I could be using the wrong keywords but I the ones I tried turned up nothing on the respective OE and Yocto mailing lists. There seem to be a huge number of pitfalls here including things like changing the DISTRO_FEATURES in effect for the images as well as selecting image specific files for packages. On a few occasions I’ve used the distro name as a way to select specific configuration files like an fstab or interfaces.

What I want is to run bitbake once for the dom0 image and have it build all the other images and install them as packages in dom0. So I’d need to have recipes that actually package the images so they can be installed in another image. I think that will be the easy part.

The hard part will be making packages specific to each image with different files specific to the image. The only thing I can come up with for this is to play ugly tricks like building each VM image with a different MACHINE type but I’m not even sure if that will work. I guess all I can do for now is to experiment a bit and get on the mailing list to make sure I’m not duplicating work that’s already been done. This could get ugly.

OpenEmbedded Xen Network Driver VM

~ flihp ~ 1 Comment

I wrote about a similar topic what feels like ages ago and I guess it was (8 months is a long time in this business). Since then I’ve been throwing some spare time at this same problem and I’ve actually made measurable progress. There have been a number of serendipitous events that came together to make this possible, the most important of which is the massive update to the Xen recipe in meta-virtualization. With this it’s super easy to crank out a Xen pvops kernel so combining this with an image that has the right plumbing in place it’s not as hard as you might think to build an NDVM.

So armed with the new Xen stuff from meta-virtualization I set out to build a reference NDVM. This isn’t intended to replace the NDVM in a system like XenClient-XT which is far more sophisticated. It’s just intended for experimentation and I don’t intend to build anything more sophisticated than a dumb Ethernet bridge into it.

To host this I’ve started a layer I call ‘meta-integral’. I know, all the good names were taken. Anyways this is intended to be as sort of distro layer where I can experiment with Xen stuff. Currently I’ve got a distro config for dom0 and an NDVM. The dom0 work is still very much a work in progress but the NDVM (much simpler) will actually boot as a PV guest.

To build this just clone my git repo with the build scripts and it’ll do all of the hard work for you:

git clone https://github.com/flihp/oe-build-scripts.git
git checkout ndvm
./build.sh | tee build.log

This will crank out an image suitable to run on an Intel SandyBridge (SNB) system. I’ve only tested PV guests so you’ll have to set up a config like the following:

kernel = "/usr/lib/xen-common/bzImage-sugarbay.bin"
extra = "root=/dev/xvda console=hvc0"
iommu = "soft"
memory = "512"
name = "ndvm"
uuid = "a9ae8853-f1e9-41ca-9904-0e906efeb4af"
vcpus = "1"

disk = ['phy:/dev/loop0,xvda,w']
pci = ['0000:04:00.0']

Notice the kernel image and the rootfs image must be copied over to the Xen dom0 that you want to test the NDVM on. The image is listed in the kernel line and this can be found at tmp-eglibc/deploy/images/sugarbay/bzImage-sugarbay.bin relative to your build root. The image will be in the same directory and called something like integral-image-ndvm-sugarbay.ext3. Notice that the disk config is pointing at a loopback. You’ll have to set this up with losetup just like any other loopback device. The part that differentiates this from any other PV guest is that we’re passing a PCI network device through to it and it’ll offer up a bridge to other guest VMs. The definitive documentation on how to do this with Xen is here: http://wiki.xen.org/wiki/Xen_PCI_Passthrough

The bit that I had to wrangle to get the bridge set up properly with OE was the integration between a network interfaces file and the bridge. I’ve been spoiled by Debian and it’s seamless integration between the two. OE has no such niceties. In this situation I had to chose between hacking a script manually or finding the scripts that integrate interfaces configuration with the bridge and baking that into the bridge-utils package from meta-oe. I figured getting bridges integrated with interfaces would be useful to others so I went through the Debian source package, extracted the scripts and baked them into OE directly. Likely this should go ‘upstream’ but for now this specialization is just sitting in my meta-integral layer.

So after fixing up the bridge-utils package so it plays nice with the interfaces file, the interfaces in our NDVM looks like so:

# /etc/network/interfaces -- configuration file for ifup(8), ifdown(8)
 
# The loopback interface
auto lo
iface lo inet loopback

# real interface
auto eth0
iface eth0 inet manual

# xen bridge
auto xenbr0
iface xenbr0 inet manual
        bridge_ports eth0
        bridge_stp off
        bridge_waitport 0
        bridge_fw 0

So that’s it. Boot up this NDVM and it’ll have a physical network device and a bridge ready for consumption by other guests. I’ve not yet gone through and tested adding additional guests to the bridge so I’m assuming there’s still a bit of work lurking there. I’ll give this last bit a go and hopefully have positive results to post sooner than later. I’ve also not tested this on XenClient-XT as the most recent stable release is getting a bit old and likely there’s going to be incompatibilities between netfront / back stuff. This approach however is likely a great starting point if you’re building a service VM you want to run on our next release of XT though so feel free to fork and experiment.

UPDATE: Gave my NDVM a test just by giving the dom0 that was hosting it a vif. You can do this like so: 

# xl network-attach Domain-0 backend=ndvm

The above assumes your NDVM has been named ‘ndvm’ in it’s VM config naturally. Anyways this will pop up a vif in dom0 backed by the NDVM. Pretty slick IMHO. Now to wrap this whole thing up so dom0 and the NDVM can be built as a single image with OE … Sounds easy enough 🙂

Using OE to build an XT ‘Service VM’

~ flihp ~ Leave a comment

UPDATE: I’ve deleted the build scripts git repo mentioned in this post and rolled all of my OE project build scripts into one repo. Find it here: git://github.com/flihp/oe-build-scripts.git

UPDATE #2: I’ve written a more up-to-date post on similar work here: http://twobit.us/blog/2013/11/openembedded-xen-network-driver-vm/. The data in this post should be considered out of date.

Over the past few weeks I’ve run into a few misconceptions about XenClient XT and OpenEmbedded. First is that XT is some sort of magical system that mere mortals can’t customize. Second is that building a special-purpose, super small Linux image on OpenEmbedded is an insurmountable task. This post is an attempt to dispel both of these misconceptions and maybe even motivate some fun work in the process.

Don’t get me wrong though, this isn’t a trival task and I didn’t start and end this work in one night. There’s still a bunch of work to do here. I’ll lay that out at the end though. For now, I’ve put up the build scripts I threw together last night on git hub. They’re super minimal and derived from another project. Get them here: https://github.com/flihp/transbridge-build-scripts

My goal here is to build a simple rootfs that XT can boot with a VM ‘type’ of ‘servicevm’. This is the type that the XT toolstack associates with the default ‘Network’ VM. Basically it will be a VM invisible to the user. Eventually I’d like for this example to be useful as a transparent network bridge suitable as an in-line filter or even as a ‘driver domain’. But let’s not get ahead of ourselves …

What image do I build?

The first thing that you need to chose to do when building an image with OE is what MACHINE you’re building for. XT uses Xen for virtualization so whatever platform you’re running it on will dictate the MACHINE. Since XT only runs on Intel hardware it’s pretty safe to assume you’re system is compatible with generic i586. The basic qemux86 MACHINE that’s in the oe-core layer builds for this so for these purposes it’ll suffice. This is already set up in the local.conf in my build scripts.

To build the minimal core image that’s in the oe-core layer just run my build.sh script from the root of the repository. I like to tee the output to a log file for inspection in the event of a failure:

./build.sh | tee build.log

Now you should have a bunch of new stuff in ./tmp-eglibc/deploy/images/ which includes an ext3 rootfs. The file name should be something like core-image-minimal.ext3. Copy this over to your XT dom0 and get ready to build a VM.

Make a VHD

The next thing to do is copy the ext3 image over to a VHD. From within /storage/disks create a new VHD large enough to hold the image. I’ve experimented with both core-image-basic and core-image-minimal and a 100M VHD will be large enough … yes that’s a very small rootfs. core-image-minimal is around 9M:

cd /storage/disks
vhd-util create -n transbridge.vhd -s 100

Next have tap-ctl create a new device node for the VHD:

tap-ctl create -a vhd:/storage/disks/transbridge.vhd

This will output the path to the device node created (and yeah the weird command syntax bugs me too). You can alternatively list the current blktap devices and find yours there:

tap-ctl list
1276    0    0        vhd /storage/ndvm/ndvm.vhd
1281    1    0        vhd /storage/ndvm/ndvm-swap.vhd
...

I’ve got no idea what the first number is (maybe PID of the blktap instance that’s backing the device?) but the 2nd and 3rd numbers are the major / minor for the device. The last column is the VHD file backing the device so find your VHD there and the major number then find the device in /dev/xen/blktap-2/tapdevX … mine had a major number of ‘8’ so that’s what I’ll use in this example. Then just byte copy your ext3 on to this device:

dd if=/storage/disks/transbridge.ext3 of=/dev/xen/blktap-2/tapdev8

Then you can mount your VHD in dom0 to poke around:

mount /dev/xen/blktap-2/tapdev8 /media/hdd

Where’s my kernel?

Yeah so OE doesn’t put a kernel on the rootfs for qemu machines. That’s part of why the core-image-minimal image is so damn small. QEMU doesn’t actually boot like regular hardware so you actually pass it the kernel on the command line so OE’s doing the right thing here. If you want the kernel from the OE build it’ll be in ./tmp-eglibc/deploy/images/ with the images … but it won’t boot on XT 😦

This is a kernel configuration thing. I could have spent a few days creating a new meta layer and customizing the Yocto kernel to get a ‘Xen-ified’ image but taht sounds like a lot of work. I’m happy for this to be quick and dirty for the time being so I just stole the kernel image from the XT ‘Network’ VM to see if I could get my VM booting.

You can do this too by first mounting the Network VMs rootfs. Cool thing is you don’t need to power down the Network VM to mount it’s FS in dom0! The disk is exposed to the Network VM as a read-only device so you can mount it read only in dom0:

mount /dev/xen/blktap-2/tapdev0 /media/cf

Then just copy the kernel and modules over to your new rootfs and set up some symlinks to the kernel image so it’s easy to find:

cp /media/cf/boot/vmlinuz-2.6.32.12-0.7.1 /media/hdd/boot
cp -R /media/cf/lib/modules/2.6.32.12-0.7.1 /media/hdd/lib/modules
cd /media/hdd/boot
ln -s vmlinuz-2.6.32.12-0.7.1 vmlinuz
cd /media/hdd
ln -s ./boot/vmlinuz-2.6.32.12-0.7.1 vmlinuz

You may find that there isn’t enough space on the ext3 image you copied on to the VHD. Remember that the ext3 image is only as large as the disk image created by OE. It’s size won’t be the same as the VHD you created unless you resize it to fill the full VHD. You can do so by first umount’ing the tapdev then running resize2fs on the tapdev:

umount /media/hdd
resize2fs /dev/xen/blktap-2/tapdev8

This will make the file system on the VHD expand to fill the full virtual disk. If you made your VHD large enough you’ll have enough space for the kernel and modules. Like I say above, 100M is a safe number but you can go smaller.

Finally you’ll want to be able to log into your VM. If you picked the minimal image it won’t have ssh or anything so you’ll need a getty listening on the xen console device. Add the following line to your inittab:

echo -e "nX:2345:respawn:/sbin/getty 115200 xvc0" >> /media/hdd/etc/inittab

There’s also gonna be a default getty trying to attach to ttyS0 which isn’t present. When the VM is up it will cause some messages on the console:

respawning too fast: disabled for 5 minutes

You can disable this by removing the ‘S’ entry in inittab but really the proper solution is a new image with a proper inittab for an XT service VM … I’ll get there eventually.

Make it a VM

Up till now all we’ve got is a VHD file. Without a VM to run it nothing interesting is gonna happen though so now we make one. The XT toolstack isn’t documented to the point where someone can just read the man page. But it will tell you a lot about itself if you just run it with out any parameters. Honestly I know very little about our toolstack so I’m always executing xec and grepping though the output.

After some experimentation here are the commands to create a new Linux VM from the provided template and modify it to be a para-virtualized service VM. In retrospect it may be better to use the ‘ndvm’ template but this is how I did it for better or for worse:

xec create-vm-with-template new-vm-linux

This command will output a path to the VM node in the XT configuration database. The name of the VM will also be something crazy. Ge the name from the output of xec-vm and change it to something sensible like ‘minimal’:

xec-vm --name  set name minimal

Your VM will also get a virutal CD-ROM which we don’t want so delete it and then add a disk for VHD we configured:

xec-vm --name minimal --disk 0 delete
xec-vm --name minimal add-disk
xec-vm --name minimal --disk 0 set phys-path /storage/disks/minimal.vhd

Then set all of the VM properties per the instructions provided in the XT Developer Guide:

xec-vm --name minimal --disk 0 set virt-path xvda
xec-vm --name minimal set flask-label "system_u:system_r:nilfvm_t"
xec-vm --name minimal set stubdom false
xec-vm --name minimal set hvm false
xec-vm --name minimal set qemu-dm-path ""
xec-vm --name minimal set slot -1
xec-vm --name minimal set type servicevm
xec-vm --name minimal set kernel /tmp/minimal-vmlinuz
xec-vm --name minimal set kernel-extract /vmlinuz
xec-vm --name minimal set cmd-line "root=/dev/xvda xencons=xvc0 console=xvc0 rw"

Then all that’s left is booting your new minimal VM:

xec-vm --name minimal start

You can then connect to the dom0 end of the Xen serial device to log into your VM:

screen $(xenstore-read /local/domain/$(xec-vm --name minimal get domid)/console/tty))

Next steps

This is a pretty rough set of instructions but it will produce a bootable VM on XenClient XT from a very small OpenEmbedded core-image-minimal. There’s tons of places this can be cleaned up starting with an kernel that’s specific to a Xen domU. A real OE DISTRO would be another welcome addition so various distro specific features could be added and removed more easily. If the lazywebs feel like contributing some OE skills to this effort leave me a comment.

Measured Launch on OE core

~ flihp ~ 2 Comments

It’s been 4 months since my last post but I’ve been working on some fun stuff. Said work has progressed to the point where it’s actually worth talking about publically so I’m crawling out from under my favorite rock and putting it “out there”.

My last few bits of writing were about some random OpenEmbedded stuff, basically outlining things I was learning while bumbling my way through the OE basics. I’ve been reading through the meta-selinux and meta-virtualization layers and they’re a great place to learn. Over the winter Holiday here I had some extra vacation time from my day job to burn so I finally got serious about a project I’ve been meaning to start for way too long.

meta-measured

Over the past year I’ve been thinking a lot about the “right way” to measure a software system. We’ve implemented a measurement architecture on XT but this has a few down sides: First a system as large as XT is very difficult to use as a teaching tool. It’s hard to explain and show someone the benefits of measuring a system when your example is large, complex and the relevant bits are spread throughout the whole system. Even our engineers who know our build system inside and out often get lost in the details. Second the code belongs to Citrix and closed source software isn’t very useful to anyone except the people selling it.

So after reading through the meta-selinux and meta-xen layers a bunch and learning a good bit about writing recipes I’ve started work on a reference image for a “measured system”. I’m keeping the recipes that make up this work in a layer I call ‘meta-measured’. For this first post on the topic of measured systems I’ll stick to discussing the basic mechanics of it’s construction. This includes some data on the supporting recipes and some of the component parts necessary for booting it. Hopefully along the way I’ll be able to justify the work by discussing the potential benefits to system security but the theory and architecture discussions will be left for a later post.

get the source

If you’re interested in just building it and playing with the live image this is where you should start. Take a look and let me know what you think. Feedback would be much appreciated.

All of the work I’ve done to get this first bootable image working is up on my github. You can get there, from here: https://github.com/flihp. The ‘meta-measured’ layer is here: https://github.com/flihp/meta-measured.git. To automate setting up a build environment for this I’ve got another repo with a few scripts to checkout the necessary supporting software (bitbake / OE / meta-intel etc), a local.conf (which you may need to modify for your environment), and a script to build the ‘iso’ that can be written to a USB drive for booting a test system: https://github.com/flihp/measured-build-scripts.

The best way to build this currently is to checkout the measured-build-scripts repo:

git clone git://github.com/flihp/measured-build-scripts.git

run the ‘fetch.sh’ script to populate the required git submodules and to clone the meta-measured layer:

cd measured-build-scripts
./fetch.sh

build the iso

If you try to run the ./build.sh script next as you would think you should, the build will fail currently. It will do so while attempting to download the SINIT / ACM module for TXT / tboot because Intel hides the ACMs behind a legal terms wall with terms that must be accepted before the files can be downloaded. I’ve put the direct link to it in the recipe but the download fails unless you’ve got the right cookie in your browser so wget blows up. Download it yourself from here: http://software.intel.com/en-us/articles/intel-trusted-execution-technology, then drop the zip into your ‘download’ directory manually. I’ve got the local.conf with DL_DIR hardwired to /mnt/openembedded/downloads so you’ll likely want to change this to suit your environment.

Anyway I’ll sort out a way to fool the Intel lawyer wall eventually … I’m tempted to mirror these files since the legal notice seems to allow this but I don’t really have the bandwidth ATM. Once you’ve got this sorted, run the build.sh script. I typically tee the output to a file for debugging … this is some very ‘pre-alpha’ stuff so you should expect to debug the build a bit 🙂

./build.sh | tee build.log

This will build a few images from the measured-image-bootimg recipe (tarballs, cpios, and an iso). The local.conf I’ve got in my build directory is specific to my test hardware so if you’ve got an Intel SugarBay system to test on then you can dump the ISO directly to a USB stick and boot it. If you don’t have a SugarBay system then you’ll have to do some work to get it booting since this measured boot stuff is closely tied to the hardware, though the ACMs I’ve packaged work for 2nd and 3rd gen i5 and i7 hardware (Sandy and Ivy Bridge).

recipes

I’ve organized the recipes that make up this work into two categories: Those that are specific to the TPM and those that are specific to TXT / tboot. Each of these two technologies requires some kernel configs so those are separated out into fragments like I’ve found in other layers. My test hardware has USB 3.0 ports which the base OE layers don’t seem to have yet. I’ve included this config in my oe-measured distro just so I can use the ports on the front of my test system.

The TPM recipes automate building the Trousers daemon, libtspi and some user space tools that consume the TSS interface. Recipes for the TPM software are pretty straight forward as most are autotools projects. Some work was required to get the trousers project separated into packages for the daemon and library.

The tboot recipes were a bit more work because tboot packages a bunch of utilites in the main tboot source tree so they had to be separated out into different packages (this work is still on-going). Further tboot doesn’t use autotools and they squash most compiler flags that the OE environment passes in. The compler flags required by tboot are static which stands at odds with OE and a cross-compiled environment that wants to change the path to everything including the compiler.

I’ve no clue if tboot will build properly on anything other than an Intel system. Further the issue of Intel hiding the ACMs required for their chipssets behind an EULA wall is annoying as the default OE fetcher won’t work.

images

My first instinct is always to to describe a system by construction: from the bottom up. In this case I think going top-down is a better approach so we’ll start with the rootfs and work backwards. The TPM recipes includes two images based on the core-image from OE core. That’s one initramfs image and one rootfs. The rootfs is just the core-image with the TPM kernel drivers, trousers daemon, tpm-tools and the tpm-quote-tools. I haven’t done much with this rootfs other than booting it up and see if TXT and the TPM works as expected.

There’s also an initramfs with the TPM kernel drivers, trousers daemon and the tpm-tools but not the quote tools. This is a very minimal initramfs with the TSS daemon loaded manually in the initrd script. It’s not expected that users will be using the tpm-tools interactively here but that’s what I’ve been doing for initial testing. Only the tpm_extendpcr tool (open source from Citrix) is used to extend a PCR with the sha1sum hash of the rootfs before the call to switch_root. This requires that the ‘coreutils’ package be included just for the one utility which bloats the initramfs unfortunately. Slimming this down should’t be too much work in the future. Anyway I think this is ‘the right way’ to extend the measurement chain from the initramfs up to the rootfs of the system.

The rest of the measruements we care about are taken care of by the components from the TXT recipes. There’s only one image in the TXT recipe group however. This is derived from the OE core live image and it’s intended to be ‘deployable’ in the lanugage of OE recipes. I think this means an hddimg or an ISO image, basically something you can ‘dd’ to disk and boot. Currently it’s the basis for a live image but could easily be used for something like an installer simply by switching out the rootfs.

This image is not a separate root filesystem but instead it’s an image created with the files necessary to boot the system: syslinux (configured with the mboot.c32 comboot module), tboot, acms and the initrd and the rootfs from the TPM recipes. tboot measures the bootloader config, all of the boot modules and a bunch of other stuff (see the README in the tboot sources for details). It stores these measurements in the TPM for us, creating the ‘dynamic root of trust for measurement’ (DRTM).

Once tboot has measured all of the modules, the initramfs takes over. The initramfs then measures the rootfs as described above before the switch to root. I’ve added a few kernel parameters to pass the name of the rootfs and the PCR where it’s measurement is to be stored.

If the rootfs is measured on each boot it must be mounted read-only to prevent its measurement from changing … yup even mounting a journaled file system read-write will modify the journal and change the filesystem. Creating a read-only image is a bit of work so for this first prototype I’ve used a bit of a short cut: I’ve mounted the rootfs read only, create a ramfs read write, then the two are combined in a unionfs. In this configuration when rootfs boots it looks like a read / write mount. Thus on each boot the measurements in the TPM are the same.

Next Steps

Measuring a system is all well and good but who cares? Measurements are only useful when they’re communicated to external parties. For now this image only takes measurements and these measurements are the same on each boot. That’s it. Where this can be most immediately useful is that these measurements can be predicted in the build.

The PCRs 0-7 are reserved for the BIOs and we have no way of predicting these values currently as they’re unique to the platform and that’s messy. The tboot PCRs however (17, 18 and 19 in the Legacy mapping currently used) can be calculated based on the hashing done by tboot (read their docs and http://www.mail-archive.com/tboot-devel@lists.sourceforge.net/msg00069.html). The PCR value containing the measurement of the rootfs can be calculated quite simply as well.

For a reference live image this is interesting only in an academic capacity. As I suggest above, this image can be used as a template for something like an installer which would give the predictability of PCR values much deeper meaning: Consider an installer architecture where the installer itself is a very small rootfs that downloads the install package from a remote server (basically Debian’s netboot iso or a PXE boot setup). Assuming we have a method for exchanging system measurements (more future work) it would be very useful for the remote server to be able to evaluate measurements from the installer before releasing the install package.

This is probably a good place to wrap up this post. The meta-measured layer I’ve described is still very new and the images I’ve built are still usefuly only for ‘tire-kicking’. My next post will hopefully discuss predicting measurement values in the build system and other fun stuffs.