Raligh Restoration Update #2

My last post about this restoration project was a pretty big one. I’m thinking it’s a better idea to post about my progress in smaller steps and thus this post will be much shorter. Since last time I’ve taken on the brakes: they’ve been disassembled, cleaned, polished and greased. Here’s the before shot of the disassembled front brake:

After cleaning, polishing and reassembling the front brake here’s a shot of it beside the rear brake before cleaning:

Finally here’s the glamor shot of both cleaned and polished brakes:

I’m really pleased with how effective Mothers polish was on these. It’s kinda funny but it took me a little bit to get my technique down so the front brake isn’t quit as shiny as the rear (which I did second). No chance I’m taking it apart to polish it again though (my OCD isn’t that bad yet).

I’ve also got some new parts: a headset to replace the previous one that was completely shot, new pedals and some toe clips too. I’ll put up pictures and details as I put the bike back together.

Racoon IPsec VPN on Debian Lenny

I’ve been wanting to set up a “pure” IPsec VPN using racoon for a while now. Part just for fun, part because I can. I spent a weekend on it once a while back, didn’t make much progress, got sick of trying to figure out cryptic racoon debug output and then gave up (more pressing stuff, you know how it is).

Anyways I ran into a situation where I NEED a VPN now. I manage a few systems that are facing the internet so they’re constantly under attack (mostly bots trying to brute force ssh all the time). Remote administration over ssh is pretty much all I need but I’d like to be able to keep a closer eye on the hardware through the “Integrated Lights-Out” system (they’re HP Proliant servers). These I don’t want facing the internet. Similarly I don’t want the configuration interfaces for my switch facing the public either.

So what I needed was a management network that I could connect to through a VPN gateway remotely (typically known as a “roadwarrior” setup). The ALIX system I’ve been blogging about in the recent past is what I’m using as the gateway / server. This post is a quick run down of the contortions I went through to get this working and why I didn’t get it working just how I want it 😦


  • racoon only, no L2TP
  • rsasig authentication
  • as little hard coded network configuration as possible on the client

I thought the above was pretty ambitious. Configuring racoon is pretty complicated but after pouring over the man page for racoon.conf I found the mode_cfg section which specifies the network configuration for the server to send out to authenticated clients. A little more digging turned up a few examples, particurlarly useful were the netbsd howto and the howto forge racoon roadwarrior configuration.

Both of these give a working example of using the hybrid_rsa authentication with mode_cfg. This isn’t exactly what I wanted but it solves 2 of my 3 requirements above so it’s a great start. Next was adding my own server and client certificates to the configuration and making sure that both the certificate and the remote identifier were being verified. I didn’t want to keep having to type in a password when connecting to the VPN so I moved on to getting rsasig authentication working. Naturally at this point all hell broke loose.

It took me forever to figure it out, but it looks like the ipsec-tools version that ships with Lenny (0.7.1) doesn’t play nice with rsasig authentication and mode_cfg. The client and server are able to negotiate phase 1 without any troubles when the client never requests the configuration data from the server. I tried all sorts of configuration combinations hoping to find something that worked with no luck. Eventually I ran across an old and unanswered post to the ipsec-tools users mailing list from a few years back describing the same problem I’m having with the 0.7.0 version of racoon. Probably safe to assume that this behavior is what I’m running into on the version 0.7.1.

At this point my options were to upgrade to a later version and hope the bug was fixed or use hybrid auth. Guess which one I chose … hybrid auth ain’t so bad 🙂 Yeah typing in a password is a PITA but both client and server can still be configured to check each others certs and asn1dns identifiers in phase 1 so very little (if any) security is compromised. 2 out of 3 requirements isn’t bad. None of the desired functionality was lost but I do have to supply a password each time I connect to the VPN. Meh.


Since the articles on howto forge and netbsd.org are so good I won’t bore you with a full description of my racoon configuration since it’s very similar. I will include them here for completeness and cover the parts where they differ.

The server.racoon.conf is very similar except for the credential verification for the client and some details in the mode_cfg. I’m using the split_network directive to send routing information to the client so we don’t have to hardcode any routes. The scripts on the client side had to be changed to accommodate this but it wasn’t that hard (I’ll get to this in a second). Also notice that I’m using all class C networks (/24 in CIDR) so no routes need to be specified on systems plugged into the management network directly.

In my client.racoon.conf the only difference is in the verification of the servers credentials (certs). I started out testing using certificates generated from my own root CA. When this was deployed I got certificates from cacert.org which is a great service.

The significant changes on the client side were in the phase one up and down networking scripts. I really didn’t like the scripts from the howto forge article (some of it didn’t seem necessary). They were a great starting place though but since I’m using the split_network I had to be able to properly handle dynamically creating and removing these routes. The final scripts can be found here: client.phaseone-up.sh and client.phaseone-down.sh

If you look closely you’ll notice that in the phaseone-down script I flush all SAs when taking down the VPN. Obviously this isn’t what we want: we want only to delete the relevant SAs. Unfortunately version 0.7.1 of the ipsec-tools package doesn’t play well with the way Linux manages SAs so the deleteall setkey function doesn’t work right. Flush is all we’re left with unless we’re gona parse the SADB output for the SPIs we need. This bug was reported on the ipsec-tools devel mailing list with a patch and it seemed well received so it’s likely in a later release. I’ll put off writing fancy bash script and just upgrade racoon soon.

Hope this is useful to everyone out there setting up a roadwarrior friendly racoon VPN. Leave me a comment if this was useful or if any of it’s unclear.

Raligh Restoration Update #1

The last post I made about this winter project was a while ago. Since then I’ve made some great progress. So far I’ve only focused on cleaning up the easy stuff: getting rust off of the wheels, handle bars and neck, and the break and gear leavers. All the grunt work that no one likes doing.

Wheels and Hubs

I started off with the wheels since I figured they would be the easiest. A friend recommended that I use Navel Jelly (Phosphoric Acid), a powerful rust remover. The warning to keep from introducing this stuff directly into surface water was enough to keep me away though since I’m doing this in my basement and can only dispose of stuff down the drain. Instead I took the elbow-grease approach and bought a few packs of green scouring pads and went to work.

For those interested, Phosphoric acid is what makes Coke-a-Cola a good rust remover. When we were kids we’d soak rusty bike parts in coke to loosen up the rust before scrubbing them down. I never knew why it worked so well. Now I know. Read the Wikipedia article above for more things that Coke is good for like decreasing bone density etc.

After about 6 hours of scrubbing, a half dozen scrub pads and a few layers of skin off both of my hands the wheels looked pretty good. I took them both to the new bike shop down the street from me called Mello Velo to have Steve true them and rebuild the hubs (truing wheels isn’t something I’ve gotten into yet). The end result looks pretty good. Here are some before / after shots:


Probably should have done more of a side-by-side for each hub but I’m learning this as I go. I’m really pumped about how well they turned out.

Headset, Handlebars and Leavers

Next I took apart the headset and removed the fork, neck, handlebars and brake leavers. The headset and handlebars cleaned up super easy. The only hard part was getting into some of the tight spots around the neck. A small stiff wire brush worked OK. The same green scouring pads did the trick on the rest of it:

The shift levers came out well though they are pitted in a few spots. Don’t think there’s much that can be done about this though.
After cleaning up the levers I put a protective coat of grease on all the metal bits.

The brake leavers and the brakes themselves are aluminum so these were just dull, no rust. I took some Mothers polish to the leavers and they shined up pretty good. Here’s a side by side of the two levers. The one on the left has been polished, the leaver on the right hasn’t been cleaned yet. That’s not the lighting that makes them look like they’re not the same color. After polishing they feel super smooth and got nice and shiny. They’re still pretty beat up in a few spots where the previous owner took a spill or two but they look much better.

I also popped out the cups and bearings for the headset. They were way beyond repair as expected so I’ve got a new set on order. I got a chance to use my headset race remover for the intended purpose finally. I bought it for popping pressed bearings out of the bottom bracket on my BMX bikes. Looks like they work just as well on headsets.

Taking a look at the frame without the headset gives a good indication that things are headed in the right direction:

I haven’t done anything to the front fork or the front brakes yet. The brakes are aluminum so they’ll get the same treatment as the levers. I don’t think the fork needs anything beyond some soap and water. There’s a good bit of paint missing but it’s doesn’t look bad, just well used / loved. That’s right, if your bike doesn’t have a few scratches you obviously don’t love it. When I get bored maybe I’ll paint it but that’s a long way off.

Here’s another fun one: The complete set of tools and parts. Notice the Park Tools bottle opener. No tool box is complete without it.

More to come soon.

Amazon mp3 Downloader Woes

First if you’re new to the Amazon MP3 downloader and you’re running Debian check out the howto on the Ubuntu forums. It’s very helpful.

Now on to my complaints (that’s what blogs are for right?)

I think it’s pretty cool that Amazon released their MP3 downloader program for Linux. That’s about the only nice thing I have to say about it. Well not really. They package it for Debian and a whole bunch of other distros so they get credit for that too.

My first gripe is that they only distribute a 32 bit x86 version. Compiling and packaging this for x86_64 (amd64) would take such a small amount of effort on their part it isn’t funny. On the client side installing a 32 bit application is a real pain. You’ve gotta stumble through downloading the 32 bit libraries by hand. Utilities like getlibs make it much less painful but it’s still not nearly as good as the dependency tracking that are built into .deb files for a reason.

But the reason I’m writing this is that, after about 3 months using this application without any problems it broke this week. After buying 2 albums it started giving me an error: “Can’t connect. Please check your internet connection…” That’s just insulting guys. I’ve got Pandora streaming, 2 ssh sessions, an imap and a VPN connection open and you’re gona tell me to check my internet connection. Right.

So after mucking around for a while it turns out this is due to a dependency on the lib32nss-mdns package. Their deb package didn’t have lib32nss-mdns as a dependency so how was I supposed to know?. Even stranger is how suddenly after 2 months of using the downloader (and probably $200 spent on music) it suddenly stopped working. Why’d it work before? My guess is they updated (aka broke) something on the server side and since they don’t expose their application through an apt repository there was no way to notify users except by breaking the client application.

After finally figuring out what’s wrong I just went ahead and downloaded the new version of the Amazon MP3 client … just to find out that a few failed attempts to download your purchase will cause Amazon to lock you out. That’s right, I can’t download my MP3s because they broke their client. Now I’ve gotta go through customer service and ask Amazon to unlock my music. What a joke.

But there is hope: there’s a command line client for the Amazon mp3 store that’s opensource. I haven’t tried it yet but if this thing breaks again I’ll make the switch.

Using tmpfs to Minimize Disk IO

Now that I’ve got my ALIX system up and running Lenny, it’s time to tweak the configuration. One of the things I liked best about the Voyage distribution is its use of tmpfs for the directories that receive a lot of writes to minimize the IO on the compact flash (CF) card. The reason for doing this is there’s a maximum number of write cycles that can be made to the CF card. Not that I’ve actually worn out a CF card before but I don’t intend to either.

I want to have /tmp, /var/run, /var/lock and /var/log mounted as tmpfs. There’s a few resources out there that provide scripts and methods for doing this but I’m not a big fan of any of them (see references section below). Debian has almost all of the necessary machinery to perform this task with minimal custom scripting. We’ll be be mucking around in the /etc/init.d and /etc/rcS directories but as little as possible.

/var/run and /var/lock

A significant portion of what we want can be achieved using the features of the mountkernfs.sh script. There are two variables called RAMRUN and RAMLOCK that control whether or not /var/run and /var/lock are mounted as tmpfs respectively. These variables are set in /etc/default/rcS and the mount points are created in the /etc/init.d/mountkernfs.sh script if the associated variable is set to “yes”.

There does seem to be a small bug in this script however. It does not import the variables it needs from /etc/defaults/rcS. I’m pretty sure this is a bug and can be fixed with a very small patch

--- ./mountkernfs.sh.old	2010-01-02 22:32:44.000000000 -0500
+++ ./mountkernfs.sh	2010-01-02 22:33:09.000000000 -0500
@@ -18,6 +18,7 @@
 . /lib/init/mount-functions.sh
 [ -f /etc/default/tmpfs ] && . /etc/default/tmpfs
+[ -f /etc/default/rcS ] && . /etc/default/rcS
 do_start () {

/tmp and /var/log

After this we’re half way to achieving our goal. It would be nice if the /var/log directory could be mounted as easily but most people will tell you that having log files reside on non-persistent storage is a very bad idea. If something goes wrong and your system goes down you won’t be able to analyze your log files. This is a very real concern which we will address shortly. First the remaining two mount points need to be mounted through /etc/fstab with the following two entries:

tmpfs  /tmp     tmpfs   defaults,noexec,nosuid,mode=1777         0   0
tmpfs  /var/log tmpfs   defaults,noexec,nosuid,nodev,mode=755  0   0

This solves the issue of mounting /tmp but /var/log requires a little more work. Debian (and LInux in general I think) expects that some files and directories will exist in the logging directory. To account for this, after the mount scripts run we want to create the necessary file structure. I’ve done this by creating a tar archive of the expected structure and extract it to the newly mounted tmpfs /var/log directory on each system boot. The following script: logskel.sh.gz does exactly this:

. /lib/init/vars.sh
. /lib/lsb/init-functions

# get configuration info for this script
[ -e /etc/default/log-skel ] && . /etc/default/log-skel

case "$1" in
		log_begin_msg $@
                # select defaults if the configured options don't make sense
		[[ ! -f $SKEL ]] && SKEL=/lib/init/log-skel.tar.gz
		[[ ! -d $LOG_DIR ]] && LOG_DIR=/var/log
		/bin/tar -zxf ${SKEL} -C ${LOG_DIR} 2>&1 > /dev/null
		log_end_msg $?
		echo "Error: argument '$1' not supported" >&2
		exit 3
		# No-op
		echo "Usage: $NAME [start|stop]" >&2
		exit 3

You’ll need to put the archive that’s being extracted into /lib/init or specify a different location through the /etc/default/log-skel file. I’m using this structure on a system with very few daemons running: log-skel.tar.gz. You may want to build one specific to your systems needs. The above script should be run after all file systems are mounted. On a Lenny system this is done by linking the script to /etc/rcS.d/S36log-skel.sh

Persistent logging of “serious” errors

Finally we still want to log “serious” error messages from syslog to persistent storage so they aren’t lost if the system reboots. This is a single rsyslog rule that can be put in the rsyslog.conf file directly or a separate file in the /etc/rsyslog.d directory. I chose the latter: persistent.conf

*.err    /var/persistent.log

Now cross your fingers and reboot. Any messages you see during boot indicating missing log files can be fixed by adding the file to the template archive we extract in the init script above. After a successful reboot you should be able to see that these four directories are tmpfs mount points by executing the mount command. This is the full output on my ALIX system.

/dev/hda2 on / type ext2 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,nosuid,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
procbususb on /proc/bus/usb type usbfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
tmpfs on /tmp type tmpfs (rw,mode=1777)
tmpfs on /var/log type tmpfs (rw,noexec,nosuid,nodev,mode=755)

We’re interested in lines 5, 6, 11 and 12. Success!


Installing Lenny on ALIX 2d3 over Serial Console

In my last post I linked to the howto forge article that gives detailed instructions for installing Debian on a PCEngines WRAP system. After playing with Voyage Linux on my new ALIX system (the successor to the WRAP) I decided that I would be better of going with the stock Debian Lenny (5.0). The cool thing is, I didn’t follow the howto 🙂 Instead I decided to exercise the PXE-boot support in the ALIX bios and learn something new in the process. This system will be a VPN gateway to a management network that I need to access remotely.

The install requires two connections between my laptop (itself running Lenny) and the ALIX system. First the ethernet connection between the two for the PXE-boot and subsequent network install and a null-modem / serial cable to act as a console for the installer. That’s right, no VGA on this thing … old school. Here’s what it looks like:
The red box is just the housing for a retractable ethernet cable.

First off get minicom up and running. My laptop has no serial port so I broke out my new fangled USB one. Grab the ALIX manual and look up the default comport settings: 38400 8N1, flow control = none. I had a problem with minicom having the following failed assertion:

minicom "Assertion `inptr - bytebuf > (state->__count & 7)' failed"

It seems the way to solve this is by setting the LANG environment variable to “C”. You can do this by executing minicom like this:

LANG=C minicom -c on

Plug the ALIX board in now and you should see the BIOS initializing then failing to find a disk to boot from. Reset it and this time when the BIOS is performing the memory test press the ‘S’ key to enter the BIOS settings. Change the serial baud to 9600 and while you’re in the menu enable PXE-boot by pressing ‘E’.

I recommend you change the baud setting because all documentation I found on installing Linux using a serial console used this baud rate and I wanted to keep things consistent. Likely you can chose any rate you want as long as you’re consistent in the settings you chose … YMMV

Next we track down the directions for installing Debian using netboot. This is well documented on the Debian websites but naturally there are a few catches. I’ll cover those here. Specifically netboot doesn’t support serial console installs. First we’ll worry about getting PXE-boot going, then worry about the installer.

I used the tftpd-hpa tftp server as recommended and the CMU bootp server. inetd.conf already had the necessary configuration lines for these two servers, they only need to be uncommented:

tftp           dgram   udp     wait    root  /usr/sbin/in.tftpd /usr/sbin/in.tftpd -s /var/lib/tftpboot
bootps          dgram   udp     wait    root    /usr/sbin/bootpd        bootpd -i -t 120

Take note of the root directory for the tftp server. This had me scratching my head for a while. The ‘-s’ option on the first line is the root directory for the tftp server (see the man page for more). This is where we extract the the netboot.tar.gz archive. This affects the configuration we’ll use for bootpd. Note the hd option:

# /etc/bootptab: database for bootp server (/usr/sbin/bootpd)

hd is set to / since we’ve told the tftp server that /var/lib/tftpboot is it’s root directory. ha needs to be the MAC address of the NIC on the ALIX board you’re using.

Next comes the patch to add serial console support to the syslinux configuration used in the netboot. The lack of serial console in the installer is documented in bug 309223. There’s a patch posted as a work around but it’s for the amd64 installer and has a lot of options we don’t need (the GTK installer won’t do us much good over the serial console). The patch isn’t short so I won’t include it in it’s entirety. It can be downloaded here: installer.diff.gz. Copy this file to the root of the installer directory and apply the patch:

zcat installer.diff.gz | patch -p1

Notice that we’ve set the serial console to 9600 baud just like we did in the ALIX BIOS menu.

From here the installer should work just like it would using VGA. The serial console is slower (though we may be able to speed it up a bit using a higher baud rate) and the Geode CPU is only 500Mhz but the install didn’t take long. Now the last detail: I’m using my laptop to NAT traffic from the ALIX system to my wireless network when doing the install. This isn’t a requirement and if you’ve got a wired network available then you may want to just use that as is.

Next we need to configure some odds and ends specific to the ALIX system. That’s coming up next.