MAC SEED Lab Requirements

It’s been way to long since my last post on this subject. I’ve been rolling around ideas in my head for the elements that will make this Lab good, or bad for that matter. I’m just going to dump them here and refine them as I either as I run across new requirements or I realize that something on this list is a bad idea.

  • The lab task should focus on policy development and what it means to the system as a whole. Integrity and secrecy should both be addressed as part of the lab.
  • Set up of the lab should take no effort on the part of the student. SELinux should already be installed with a known good policy on their systems. The only thing they should be concerned with is writing policy, maybe some code to confine, building the policy, inserting it into the kernel and debugging the output.
  • This lab is intended to reinforce the mandatory access control concept. It’s not an SELinux lab per se. SELinux is just the MAC system used to reinforce the MAC concept. This implies that the Lab shouldn’t be bogged down in the details of managing an SELinux system. Since the labs are intended to be run from a Ubuntu VM we have to be sure SELinux is well supported or already set up on this VM.
  • Following the previous point it’s important that the MAC concepts from the class lecture be incorporated into the lab explicitly. It’s been a while since I took the class that this lab will be taught in so getting a copy of the lecture notes and ensuring I’m reinforcing the right concepts is important. I may need to make suggestions for new topics to be discussed in class but I’ll try to keep changes to the lecture minimal.
  • It would be nice if we could make practical links to a previous lab showing how MAC can defend against specific attacks. There is a lab used in this class showing buffer overflows at work. Showing the code previously developed for the buffer overflow lab thwarted by SELinux would be cool. After walking through an example this would make a good independent task for the students to undertake.
  • Simplicity. Keep the policy developed from getting too scary is a must.
  • The Reference policy by far the policy “language” to use when developing “real” policy but exposure to the raw policy is a must. It may make sense to have students determine the raw policy needed to perform a task and then have them hunt through the reference policy interfaces searching for the right interface to use. That could get ugly though. That may not even be practical since reading the reference policy requires a certain amount of skill. This ones gona take some thought.

I’m going to let these sit for a day or two and do some thinking. Refinements will follow as will a task list derived from the “final” requirements. I know, requirements are never final but I’ll pretend they are when I move on to the tasks … at least until they change. I’m interested in any comments or suggestions that the interwebs may have so let me know what you think.

Raligh update #3

It’s been a while since I’ve posted about the Raligh. I’ve been working on it but not as much as I’d like. You know how it is: school, work blah blah blah. Same old sob story, poor me. Anyways I’ve started in on the home stretch and it’s turned out to be a very long one.

The drive train on this thing is nasty. It’s the front and rear derailer, the chain rings, the cranks and the bottom bracket. That’s pretty much the hard stuff, specifically the derailers. These things have about 20 years of crap built up on them and it’s obvious that a few parts are bent. Here are the before pictures:

The front derailer:


The front chain wheels / rings:



Rear derailer:


And finally the cranks:

Those of you who’ve done this before will notice the cotter pin through the cranks. I’ll get to these in a in a bit but first the good news … the front chain rings and the kickstand cleaned up great!

So now the bad news: I got everything off the frame except for the cranks. Cleaning the 20 years worth of lubricant and dirt off the frame was a pain in the ass and there’s still more to be done but I can’t get to it all till the cranks are off. But these cranks do not want to come off. Stupid cotter pins. I’ve never seen cranks with cotter pins in them and I now know why.

I did some research and found that there are cotter pin presses out there but none are less than $50 + shipping so I rigged up my own boot-leg press. I took a 3 inch c-clamp I had laying about, cupped the out-going end of the cotter pin in a 10mm socket and used the clamp to press the cotter pin out. I left the nut on the threaded end, one turn out to keep it from completely bending over. It looked like this:

That was till the c-clamp broke. I should have seen it coming when I had to turn the clamp with a pair of 9 inch pliers. It wasn’t as dramatic as you think, I just ended up pushing the threaded bit through the plunger a bit. It sank right into the hollow part of the socket. The plunger wasn’t spinning much after that. You can see the plunger pushed through in the following pic, this thing was flat initially:

So one c-clamp down I looked for other options. I went with a solution recommended by the late great Sheldon “No Clunk” Brown. His site is the first google hit for “cottered cranks” and he has a laundry list of things he’s done to get cotter pins out of old-school European bike cranks. After soaking the cotter pin in WD-40 penetrating oil I tried his claw hammer & pipe method. I didn’t ding up the frame like I was afraid I would but I did some damage to the threaded end of the pin. It didn’t budge.

I figured this was a good time to take a break so I sat down with a whiskey and read some of Sheldon Brown’s journal entries and accounts of some of the rides and events that he attended. Really cool to read about a bike enthusiast riding around the north shore of Massachusetts just a few towns over from where I grew up. Some of the pictures made me nostalgic for New England, especially the one of his bike propped up against a beautiful New England field stone wall.

Sheldon put a wealth of information into the public domain through his website. He basically was blogging every day before a blog was a blog. He passed away but his site is still out there, complete with pictures of his kick-ass beard. His advice was a big help so, thanks man. I’m still working on way to get these pins out, hopefully it won’t come down to using the drill 🙂

dnsmasq and racoon VPN

I’ve always used the standard dhcp3 server on my local networks (non-routable IPs). Never really knew of any other options and I didn’t look for any. As the few networks I manage have gotten larger I’ve wanted my DHCP server to be able to feed information into local DNS so I don’t have to maintain hosts files or remember IP addresses. I’ve heard horror stories about configuring BIND so I figured hooking up DHCP and BIND would be way too much work for my purposes.

After some digging I ran across dnsmasq [1]. It’s a DHCP server and a DNS proxy rolled into one. As it doles out DHCP leases it populates the DNS proxy with host names, just what I need. There are a good lot of howto’s out there for setting up dnsmasq so I won’t pollute the web with another one that’s likely not as good as the others. Frankly, dnsmasq can pretty much be configured with the information contained in its well documented example config file (provided as part of the Debian package).

What I will add to the inter-tubes is how I got dnsmasq to resolve names for VPN users connected to the racoon VPN that I’ve documented in a previous post [2] without interfering with other DNS configurations on the client. This requires a few modifications to the racoon server config and the client side up/down scripts. It also takes some resolvconf magic to finish the job.

Serving DNS info to VPN Clients

The configuration required to get racoon to send DNS information to clients as part of the mode_cfg is pretty straight forward.

    dns4 10.XXX.XXX.XXX;
    default_domain "vpn.example";

That’s it. The client side up script receives these config parameters in two new environment variables: INTERNAL_DNS4_LIST and DEFAULT_DOMAIN. The INTERNAL_DNS4_LIST reflects the fact that we can include the address of more than one DNS server. In this example we’ve only got one but we write our script such that it can handle the list.

In the up script we’ve got the DNS information now but what do we do with it? I’m no expert at building a resolv.conf files by hand and I really don’t want to be. We need a way to manage multiple DNS configurations at the same time such that when we need to resolve names for hosts on the VPN network they get routed to the DNS server configuration received from racoon. Other names we want resolved by whatever DNS configuration was in place when we brought up the VPN connection. The resolvconf program (yeah bad/confusing choice of names) almost does what we need.

Client Configuration with resolvconf

The man page for resolvconf is pretty straight forward but it leaves out one specific detail. By my reading of the man page I would think to call resolvconf as follows:

  echo -e "domain ${DEFAULT_DOMAIN}nnameserver ${DNS_IP}" | resolvconf -a ${INTERFACE}

Where INTERFACE would be the name of the interface we’re talking to the VPN through.

This doesn’t actually work though. After an hour of trying multiple configurations to see what I was doing wrong I thought to look at the script that the resolvconf package installed in my /etc/network/if-up.d directory. This script takes whatever DNS info was associated with the interface either statically in the interfaces file or dynamically over DHCP and feeds it into the resolvconf program. It does something funny though. The interface name used isn’t actually that of the interface. The script appends the address family to the interface name passed into resolvconf.

I tried using this convention for the VPN configuration scripts. I appended ‘.vpn’ to the interface name (very original I know) and this time the DNS info obtained over the VPN doesn’t stomp all over the existing DNS info (the configuration my laptop got from DHCP on the local network). The small addition to the racoon up script is as follows:

RESOLVCONF=$(which resolvconf)
INTERFACE=$(ip route get ${REMOTE_ADDR} 
    | grep --only-match 'dev[[:space:]][0-9a-zA-Z]*' | awk '{print $2}')
if [ -x ${RESOLVCONF} ]; then
    INPUT=""
    for DNS in ${INTERNAL_DNS4_LIST}
    do
        INPUT=${INPUT}$(echo "nameserver ${DNS}")
    done
    echo -n -e "domain ${DEFAULT_DOMAIN}n${INPUT}" 
        | resolvconf -a "${INTERFACE}.vpn" | logger -t "phaseone-up"
fi

This is a step in the right direction but it still doesn’t work exactly as we want.

The resolv.conf file generated by resolvconf after bringing up the VPN looks like this:

nameserver 192.XXX.XXX.XXX
nameserver 10.XXX.XXX.XXX
search home.example vpn.example

Here the 192.XXX.XXX.XXX DNS server was obtained by our network interface when it was brought up using DHCP. This is the DNS server on my home network. It knows the names of devices that have registered using DHCP and when searching for a hostname that’s not qualified the suffix appended is ‘home.example’. I leave off the top level suffix to prevent the proxy from forwarding bad search requests. The 10.XXX.XXX.XXX DNS server is the one that will resolve hosts on the VPN network. Again it knows the names of devices that have registered on the VPN network using DHCP and provides the search suffix of ‘vpn.example’.

Why This Doesn’t Work

Because the home DNS server is listed before the VPN DNS server it will be queried first. When asked for a host that exists on the VPN domain the query will first be sent to the DNS server on the home.example netowrk and the query will fail. The query will fall through to the next nameserver only in the case of a timeout or an error so the VPN DNS server will not be queried in this case and we can’t resolve names on the VPN network. If we switch their order manually we’ll be able to resolve names on the vpn.example network but attempts to resolve names on the home.example network will fail.

This situation is represented graphically here:

To make this more concrete, say we want to resolve the name for ‘bob’ (like if I were to run ‘ping bob’), a system on the home.example network. We’d expect the resolver to be smart enough to search through the two available DNS servers knowing their search domains. It could ask the vpn.example DNS server for ‘bob.vpn.example’ and if I didn’t find bob there it could then ask the DNS server on home.example for ‘bob.home.example’. If only the resolver functions in libc were this smart.
NOTE: we’d be in trouble if each network has a host named ‘bob’ but how to handle that situation is out of scope for this discussion.

For configurations that are relatively advanced we have to fall back on a DNS proxy like dnsmasq. Yes we’re already running dnsmasq as a DNS proxy on these two networks but the problem we’re running into is that the resolver on the client isn’t smart enough. The smarts we need are built into dnsmasq.

dnsmasq as a Client-side DNS Proxy

Installing dnsmasq on the client is painless. It’s already tied into the resolvconf system so its notified of changes to the reslover information but it preserves the behavior of the standard libc resolver described above. We can however statically configure dnsmasq to consult a particular DNS servers for a specific domain with one configuration line:

server=/domain/ip

For the network layout described we could add two lines to the dnsmasq.conf file to get the behavior we want:

server=/home.example/192.XXX.XXX.XXX
server=/vpn.example/10.XXX.XXX.XXX

Static configurations stink though (too easy) and with a little more work we can get the same effect with a short script:

#!/bin/sh                                                                       
# ip address from string to int                              
function inet_aton () {
    local count=3
    local int=0
    for num in $(echo $1 | sed -e 's/./ /g'); do
        let "int+=$num*256**$count"
        let "count-=1"
    done
    echo $int
}
pushd "/etc/resolvconf/run/interface/" > /dev/null
FILES=$(/lib/resolvconf/list-records | sed -e '/^lo.dnsmasq$/d')
for file in $FILES; do
    ns=$(cat $file | sed -n -e 's/^[[:space:]]*nameserver[[:space:]]+//p')
    PARAMS+="uint32:$(inet_aton $ns) "
    domain=$(cat $file | sed -n -e 's/^[[:space:]]*domain[[:space:]]+//p')
    PARAMS+="string:$domain "
done
dbus-send --system  --dest='uk.org.thekelleys.dnsmasq' 
    /uk/org/thekelleys/dnsmasq uk.org.thekelleys.SetServers $PARAMS
popd > /dev/null

For this script to make sense it’s important to know that when the resolvconf system is passed DNS information for a particular interface it makes a file with the name of the interface in /etc/resolvconf/run/interface/. This last script is placed in the directory /etc/resolvconf/update.d/. Each script in this directory is run every time the resolvconf information is changed. In the script we extract the nameserver and domain information from each file and send it to dnsmasq through the dnsmasq dbus interface (which must be enabled in dnsmasq.conf).

That’s it. Now each time we make a connection to the VPN the racoon client scripts send the VPN DNS info into resolvconf. resolvconf then runs its update.d scripts and the new script that we’ve provided takes this DNS information and sends it through to dnsmasq through a dbus interface. That was a lot of work, but now my VPN works the way I want it to. Well worth the effort IMHO.

I’m no dbus expert but I don’t really like the dnsmasq dbus interface. All functionality for manipulating the servers is packed into one function. As you can see from the above script it’s just “SetServers”. The interface would be much more effective and much easier to use if this one function were broken up into several, like an “AddServer”, “RemoveServer” etc. The full documentation for the dnsmasq dbus interface can be found here [3]. Proposing a few patches to fix this up would be a fun summer project 🙂

Mandatory Access Control SEED Lab

As part of the graduation requirements for my Masters degree, SU requires that we either complete a “masters project” or a thesis. Working full time while working towards my degree has really limited my ability to interact with the faculty and get involved with their research. This has mad the thesis option difficult since most of my research interests haven’t lined up with the interests of any professors I’ve been able to interact with. So the project seemed like the way to go especially since I’m really looking forward to graduating soon 🙂

The project that I’ve settled on is a lab that hopefully will be used by my advisor, Dr. Wenliang (Kevin) Du as part of his SEED project. He’s been using these labs in his computer security class and I noticed that, though Dr. Du teaches mandatory access control (MAC) he didn’t have a lab to make the topic concrete. I’ve been working with SELinux for a while now so making this lab it seemed a good fit for me.

It turns out that others have attempted this task in the past but have run into difficulties. I haven’t had a chance to see the previous attempts but Dr. Du couldn’t use their labs because they were very long and very complicated. After hearing this my brain starts to file it away under requirements. The lab has to be “short”: something that a graduate student in computer science / computer engineering can accomplish in a two week period (that’s actually a pretty long lab IMHO).

This initial post is just a quick introduction to the topic and to a new tag on my blog. If you’re interested in following this work, subscribe to my MACSEEDLab tag. I’ll be updating this with some brain storming soon.

Raligh Restoration Update #2

My last post about this restoration project was a pretty big one. I’m thinking it’s a better idea to post about my progress in smaller steps and thus this post will be much shorter. Since last time I’ve taken on the brakes: they’ve been disassembled, cleaned, polished and greased. Here’s the before shot of the disassembled front brake:

After cleaning, polishing and reassembling the front brake here’s a shot of it beside the rear brake before cleaning:

Finally here’s the glamor shot of both cleaned and polished brakes:

I’m really pleased with how effective Mothers polish was on these. It’s kinda funny but it took me a little bit to get my technique down so the front brake isn’t quit as shiny as the rear (which I did second). No chance I’m taking it apart to polish it again though (my OCD isn’t that bad yet).

I’ve also got some new parts: a headset to replace the previous one that was completely shot, new pedals and some toe clips too. I’ll put up pictures and details as I put the bike back together.

Racoon IPsec VPN on Debian Lenny

I’ve been wanting to set up a “pure” IPsec VPN using racoon for a while now. Part just for fun, part because I can. I spent a weekend on it once a while back, didn’t make much progress, got sick of trying to figure out cryptic racoon debug output and then gave up (more pressing stuff, you know how it is).

Anyways I ran into a situation where I NEED a VPN now. I manage a few systems that are facing the internet so they’re constantly under attack (mostly bots trying to brute force ssh all the time). Remote administration over ssh is pretty much all I need but I’d like to be able to keep a closer eye on the hardware through the “Integrated Lights-Out” system (they’re HP Proliant servers). These I don’t want facing the internet. Similarly I don’t want the configuration interfaces for my switch facing the public either.

So what I needed was a management network that I could connect to through a VPN gateway remotely (typically known as a “roadwarrior” setup). The ALIX system I’ve been blogging about in the recent past is what I’m using as the gateway / server. This post is a quick run down of the contortions I went through to get this working and why I didn’t get it working just how I want it 😦

Requirements

  • racoon only, no L2TP
  • rsasig authentication
  • as little hard coded network configuration as possible on the client

I thought the above was pretty ambitious. Configuring racoon is pretty complicated but after pouring over the man page for racoon.conf I found the mode_cfg section which specifies the network configuration for the server to send out to authenticated clients. A little more digging turned up a few examples, particurlarly useful were the netbsd howto and the howto forge racoon roadwarrior configuration.

Both of these give a working example of using the hybrid_rsa authentication with mode_cfg. This isn’t exactly what I wanted but it solves 2 of my 3 requirements above so it’s a great start. Next was adding my own server and client certificates to the configuration and making sure that both the certificate and the remote identifier were being verified. I didn’t want to keep having to type in a password when connecting to the VPN so I moved on to getting rsasig authentication working. Naturally at this point all hell broke loose.

It took me forever to figure it out, but it looks like the ipsec-tools version that ships with Lenny (0.7.1) doesn’t play nice with rsasig authentication and mode_cfg. The client and server are able to negotiate phase 1 without any troubles when the client never requests the configuration data from the server. I tried all sorts of configuration combinations hoping to find something that worked with no luck. Eventually I ran across an old and unanswered post to the ipsec-tools users mailing list from a few years back describing the same problem I’m having with the 0.7.0 version of racoon. Probably safe to assume that this behavior is what I’m running into on the version 0.7.1.

At this point my options were to upgrade to a later version and hope the bug was fixed or use hybrid auth. Guess which one I chose … hybrid auth ain’t so bad 🙂 Yeah typing in a password is a PITA but both client and server can still be configured to check each others certs and asn1dns identifiers in phase 1 so very little (if any) security is compromised. 2 out of 3 requirements isn’t bad. None of the desired functionality was lost but I do have to supply a password each time I connect to the VPN. Meh.

Configurations

Since the articles on howto forge and netbsd.org are so good I won’t bore you with a full description of my racoon configuration since it’s very similar. I will include them here for completeness and cover the parts where they differ.

The server.racoon.conf is very similar except for the credential verification for the client and some details in the mode_cfg. I’m using the split_network directive to send routing information to the client so we don’t have to hardcode any routes. The scripts on the client side had to be changed to accommodate this but it wasn’t that hard (I’ll get to this in a second). Also notice that I’m using all class C networks (/24 in CIDR) so no routes need to be specified on systems plugged into the management network directly.

In my client.racoon.conf the only difference is in the verification of the servers credentials (certs). I started out testing using certificates generated from my own root CA. When this was deployed I got certificates from cacert.org which is a great service.

The significant changes on the client side were in the phase one up and down networking scripts. I really didn’t like the scripts from the howto forge article (some of it didn’t seem necessary). They were a great starting place though but since I’m using the split_network I had to be able to properly handle dynamically creating and removing these routes. The final scripts can be found here: client.phaseone-up.sh and client.phaseone-down.sh

If you look closely you’ll notice that in the phaseone-down script I flush all SAs when taking down the VPN. Obviously this isn’t what we want: we want only to delete the relevant SAs. Unfortunately version 0.7.1 of the ipsec-tools package doesn’t play well with the way Linux manages SAs so the deleteall setkey function doesn’t work right. Flush is all we’re left with unless we’re gona parse the SADB output for the SPIs we need. This bug was reported on the ipsec-tools devel mailing list with a patch and it seemed well received so it’s likely in a later release. I’ll put off writing fancy bash script and just upgrade racoon soon.

Hope this is useful to everyone out there setting up a roadwarrior friendly racoon VPN. Leave me a comment if this was useful or if any of it’s unclear.

Raligh Restoration Update #1

The last post I made about this winter project was a while ago. Since then I’ve made some great progress. So far I’ve only focused on cleaning up the easy stuff: getting rust off of the wheels, handle bars and neck, and the break and gear leavers. All the grunt work that no one likes doing.

Wheels and Hubs

I started off with the wheels since I figured they would be the easiest. A friend recommended that I use Navel Jelly (Phosphoric Acid), a powerful rust remover. The warning to keep from introducing this stuff directly into surface water was enough to keep me away though since I’m doing this in my basement and can only dispose of stuff down the drain. Instead I took the elbow-grease approach and bought a few packs of green scouring pads and went to work.

For those interested, Phosphoric acid is what makes Coke-a-Cola a good rust remover. When we were kids we’d soak rusty bike parts in coke to loosen up the rust before scrubbing them down. I never knew why it worked so well. Now I know. Read the Wikipedia article above for more things that Coke is good for like decreasing bone density etc.

After about 6 hours of scrubbing, a half dozen scrub pads and a few layers of skin off both of my hands the wheels looked pretty good. I took them both to the new bike shop down the street from me called Mello Velo to have Steve true them and rebuild the hubs (truing wheels isn’t something I’ve gotten into yet). The end result looks pretty good. Here are some before / after shots:

DirtyHub
CleanHub
RimSidebySide
CleanWheels

Probably should have done more of a side-by-side for each hub but I’m learning this as I go. I’m really pumped about how well they turned out.

Headset, Handlebars and Leavers

Next I took apart the headset and removed the fork, neck, handlebars and brake leavers. The headset and handlebars cleaned up super easy. The only hard part was getting into some of the tight spots around the neck. A small stiff wire brush worked OK. The same green scouring pads did the trick on the rest of it:
BarsNeckClean

The shift levers came out well though they are pitted in a few spots. Don’t think there’s much that can be done about this though.
RaleighShiftClean
After cleaning up the levers I put a protective coat of grease on all the metal bits.

The brake leavers and the brakes themselves are aluminum so these were just dull, no rust. I took some Mothers polish to the leavers and they shined up pretty good. Here’s a side by side of the two levers. The one on the left has been polished, the leaver on the right hasn’t been cleaned yet. That’s not the lighting that makes them look like they’re not the same color. After polishing they feel super smooth and got nice and shiny. They’re still pretty beat up in a few spots where the previous owner took a spill or two but they look much better.
BrakeLeverClean

I also popped out the cups and bearings for the headset. They were way beyond repair as expected so I’ve got a new set on order. I got a chance to use my headset race remover for the intended purpose finally. I bought it for popping pressed bearings out of the bottom bracket on my BMX bikes. Looks like they work just as well on headsets.

Taking a look at the frame without the headset gives a good indication that things are headed in the right direction:
BareHead

I haven’t done anything to the front fork or the front brakes yet. The brakes are aluminum so they’ll get the same treatment as the levers. I don’t think the fork needs anything beyond some soap and water. There’s a good bit of paint missing but it’s doesn’t look bad, just well used / loved. That’s right, if your bike doesn’t have a few scratches you obviously don’t love it. When I get bored maybe I’ll paint it but that’s a long way off.

Here’s another fun one: The complete set of tools and parts. Notice the Park Tools bottle opener. No tool box is complete without it.
HeadsetBarsParts

More to come soon.

Amazon mp3 Downloader Woes

First if you’re new to the Amazon MP3 downloader and you’re running Debian check out the howto on the Ubuntu forums. It’s very helpful.

Now on to my complaints (that’s what blogs are for right?)

I think it’s pretty cool that Amazon released their MP3 downloader program for Linux. That’s about the only nice thing I have to say about it. Well not really. They package it for Debian and a whole bunch of other distros so they get credit for that too.

My first gripe is that they only distribute a 32 bit x86 version. Compiling and packaging this for x86_64 (amd64) would take such a small amount of effort on their part it isn’t funny. On the client side installing a 32 bit application is a real pain. You’ve gotta stumble through downloading the 32 bit libraries by hand. Utilities like getlibs make it much less painful but it’s still not nearly as good as the dependency tracking that are built into .deb files for a reason.

But the reason I’m writing this is that, after about 3 months using this application without any problems it broke this week. After buying 2 albums it started giving me an error: “Can’t connect. Please check your internet connection…” That’s just insulting guys. I’ve got Pandora streaming, 2 ssh sessions, an imap and a VPN connection open and you’re gona tell me to check my internet connection. Right.

So after mucking around for a while it turns out this is due to a dependency on the lib32nss-mdns package. Their deb package didn’t have lib32nss-mdns as a dependency so how was I supposed to know?. Even stranger is how suddenly after 2 months of using the downloader (and probably $200 spent on music) it suddenly stopped working. Why’d it work before? My guess is they updated (aka broke) something on the server side and since they don’t expose their application through an apt repository there was no way to notify users except by breaking the client application.

After finally figuring out what’s wrong I just went ahead and downloaded the new version of the Amazon MP3 client … just to find out that a few failed attempts to download your purchase will cause Amazon to lock you out. That’s right, I can’t download my MP3s because they broke their client. Now I’ve gotta go through customer service and ask Amazon to unlock my music. What a joke.

But there is hope: there’s a command line client for the Amazon mp3 store that’s opensource. I haven’t tried it yet but if this thing breaks again I’ll make the switch.

Using tmpfs to Minimize Disk IO

Now that I’ve got my ALIX system up and running Lenny, it’s time to tweak the configuration. One of the things I liked best about the Voyage distribution is its use of tmpfs for the directories that receive a lot of writes to minimize the IO on the compact flash (CF) card. The reason for doing this is there’s a maximum number of write cycles that can be made to the CF card. Not that I’ve actually worn out a CF card before but I don’t intend to either.

I want to have /tmp, /var/run, /var/lock and /var/log mounted as tmpfs. There’s a few resources out there that provide scripts and methods for doing this but I’m not a big fan of any of them (see references section below). Debian has almost all of the necessary machinery to perform this task with minimal custom scripting. We’ll be be mucking around in the /etc/init.d and /etc/rcS directories but as little as possible.

/var/run and /var/lock

A significant portion of what we want can be achieved using the features of the mountkernfs.sh script. There are two variables called RAMRUN and RAMLOCK that control whether or not /var/run and /var/lock are mounted as tmpfs respectively. These variables are set in /etc/default/rcS and the mount points are created in the /etc/init.d/mountkernfs.sh script if the associated variable is set to “yes”.

There does seem to be a small bug in this script however. It does not import the variables it needs from /etc/defaults/rcS. I’m pretty sure this is a bug and can be fixed with a very small patch

--- ./mountkernfs.sh.old	2010-01-02 22:32:44.000000000 -0500
+++ ./mountkernfs.sh	2010-01-02 22:33:09.000000000 -0500
@@ -18,6 +18,7 @@
 . /lib/init/mount-functions.sh
 
 [ -f /etc/default/tmpfs ] && . /etc/default/tmpfs
+[ -f /etc/default/rcS ] && . /etc/default/rcS
 
 do_start () {
 	#

/tmp and /var/log

After this we’re half way to achieving our goal. It would be nice if the /var/log directory could be mounted as easily but most people will tell you that having log files reside on non-persistent storage is a very bad idea. If something goes wrong and your system goes down you won’t be able to analyze your log files. This is a very real concern which we will address shortly. First the remaining two mount points need to be mounted through /etc/fstab with the following two entries:

tmpfs  /tmp     tmpfs   defaults,noexec,nosuid,mode=1777         0   0
tmpfs  /var/log tmpfs   defaults,noexec,nosuid,nodev,mode=755  0   0

This solves the issue of mounting /tmp but /var/log requires a little more work. Debian (and LInux in general I think) expects that some files and directories will exist in the logging directory. To account for this, after the mount scripts run we want to create the necessary file structure. I’ve done this by creating a tar archive of the expected structure and extract it to the newly mounted tmpfs /var/log directory on each system boot. The following script: logskel.sh.gz does exactly this:

PATH=/sbin:/bin
. /lib/init/vars.sh
. /lib/lsb/init-functions

# get configuration info for this script
[ -e /etc/default/log-skel ] && . /etc/default/log-skel

case "$1" in
	start|"")
		log_begin_msg $@
                # select defaults if the configured options don't make sense
		[[ ! -f $SKEL ]] && SKEL=/lib/init/log-skel.tar.gz
		[[ ! -d $LOG_DIR ]] && LOG_DIR=/var/log
		/bin/tar -zxf ${SKEL} -C ${LOG_DIR} 2>&1 > /dev/null
		log_end_msg $?
		;;
	restart|reload|force-reload)
		echo "Error: argument '$1' not supported" >&2
		exit 3
		;;
	stop)
		# No-op
		;;
	*)
		echo "Usage: $NAME [start|stop]" >&2
		exit 3
		;;
esac

You’ll need to put the archive that’s being extracted into /lib/init or specify a different location through the /etc/default/log-skel file. I’m using this structure on a system with very few daemons running: log-skel.tar.gz. You may want to build one specific to your systems needs. The above script should be run after all file systems are mounted. On a Lenny system this is done by linking the script to /etc/rcS.d/S36log-skel.sh

Persistent logging of “serious” errors

Finally we still want to log “serious” error messages from syslog to persistent storage so they aren’t lost if the system reboots. This is a single rsyslog rule that can be put in the rsyslog.conf file directly or a separate file in the /etc/rsyslog.d directory. I chose the latter: persistent.conf

*.err    /var/persistent.log

Now cross your fingers and reboot. Any messages you see during boot indicating missing log files can be fixed by adding the file to the template archive we extract in the init script above. After a successful reboot you should be able to see that these four directories are tmpfs mount points by executing the mount command. This is the full output on my ALIX system.

/dev/hda2 on / type ext2 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,nosuid,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
procbususb on /proc/bus/usb type usbfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
tmpfs on /tmp type tmpfs (rw,mode=1777)
tmpfs on /var/log type tmpfs (rw,noexec,nosuid,nodev,mode=755)

We’re interested in lines 5, 6, 11 and 12. Success!

References

Installing Lenny on ALIX 2d3 over Serial Console

In my last post I linked to the howto forge article that gives detailed instructions for installing Debian on a PCEngines WRAP system. After playing with Voyage Linux on my new ALIX system (the successor to the WRAP) I decided that I would be better of going with the stock Debian Lenny (5.0). The cool thing is, I didn’t follow the howto 🙂 Instead I decided to exercise the PXE-boot support in the ALIX bios and learn something new in the process. This system will be a VPN gateway to a management network that I need to access remotely.

The install requires two connections between my laptop (itself running Lenny) and the ALIX system. First the ethernet connection between the two for the PXE-boot and subsequent network install and a null-modem / serial cable to act as a console for the installer. That’s right, no VGA on this thing … old school. Here’s what it looks like:
ALIX-Serial
The red box is just the housing for a retractable ethernet cable.

First off get minicom up and running. My laptop has no serial port so I broke out my new fangled USB one. Grab the ALIX manual and look up the default comport settings: 38400 8N1, flow control = none. I had a problem with minicom having the following failed assertion:

minicom "Assertion `inptr - bytebuf > (state->__count & 7)' failed"

It seems the way to solve this is by setting the LANG environment variable to “C”. You can do this by executing minicom like this:

LANG=C minicom -c on

Plug the ALIX board in now and you should see the BIOS initializing then failing to find a disk to boot from. Reset it and this time when the BIOS is performing the memory test press the ‘S’ key to enter the BIOS settings. Change the serial baud to 9600 and while you’re in the menu enable PXE-boot by pressing ‘E’.

I recommend you change the baud setting because all documentation I found on installing Linux using a serial console used this baud rate and I wanted to keep things consistent. Likely you can chose any rate you want as long as you’re consistent in the settings you chose … YMMV

Next we track down the directions for installing Debian using netboot. This is well documented on the Debian websites but naturally there are a few catches. I’ll cover those here. Specifically netboot doesn’t support serial console installs. First we’ll worry about getting PXE-boot going, then worry about the installer.

I used the tftpd-hpa tftp server as recommended and the CMU bootp server. inetd.conf already had the necessary configuration lines for these two servers, they only need to be uncommented:

tftp           dgram   udp     wait    root  /usr/sbin/in.tftpd /usr/sbin/in.tftpd -s /var/lib/tftpboot
bootps          dgram   udp     wait    root    /usr/sbin/bootpd        bootpd -i -t 120

Take note of the root directory for the tftp server. This had me scratching my head for a while. The ‘-s’ option on the first line is the root directory for the tftp server (see the man page for more). This is where we extract the the netboot.tar.gz archive. This affects the configuration we’ll use for bootpd. Note the hd option:

# /etc/bootptab: database for bootp server (/usr/sbin/bootpd)
mgmtvpn:
  hd=/:
  bf=pxelinux.0:
  ip=10.1.0.1:
  sm=255.255.255.0:
  sa=10.1.0.2:
  ha=XXXXXXXXXXXX:

hd is set to / since we’ve told the tftp server that /var/lib/tftpboot is it’s root directory. ha needs to be the MAC address of the NIC on the ALIX board you’re using.

Next comes the patch to add serial console support to the syslinux configuration used in the netboot. The lack of serial console in the installer is documented in bug 309223. There’s a patch posted as a work around but it’s for the amd64 installer and has a lot of options we don’t need (the GTK installer won’t do us much good over the serial console). The patch isn’t short so I won’t include it in it’s entirety. It can be downloaded here: installer.diff.gz. Copy this file to the root of the installer directory and apply the patch:

zcat installer.diff.gz | patch -p1

Notice that we’ve set the serial console to 9600 baud just like we did in the ALIX BIOS menu.

From here the installer should work just like it would using VGA. The serial console is slower (though we may be able to speed it up a bit using a higher baud rate) and the Geode CPU is only 500Mhz but the install didn’t take long. Now the last detail: I’m using my laptop to NAT traffic from the ALIX system to my wireless network when doing the install. This isn’t a requirement and if you’ve got a wired network available then you may want to just use that as is.

Next we need to configure some odds and ends specific to the ALIX system. That’s coming up next.