Validating IP Addresses

UPDATE: Added terminating ‘$’ in ipv4 regex as noted in comment from raorn.

I’ve been working on a fix to a system script that passes around and manipulates IP addresses. With IPv6 becoming more prevalent this script must work with IPv6 addresses not just v4. While working on this and digging around the web I ran across some stuff that I think is worth sharing.

The first thing I always do when I’m working with a new data format is writing a script / function that can be used to validate it. Here’s what I came up with for IPv4 and IPv6.

IPv4 Regex

With IPv4 this pretty boring and can be done with a one line regular expression (regex) that’s all over the web. I clean things up a bit by using shell variables but the regex should be clear:

#!/bin/sh
QUAD="25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9]"
is_ipv4 () {
    echo $1 | grep --silent "^(${QUAD})(.(${QUAD})){3}$"
    if [ $? -eq 0 ]; then
        return 1
    fi
    return 0
}

is_ipv4 $1
if [ $? -eq 1 ]; then
    exit 0
else
    echo "Invalid IPv4 address." >&2
    exit 1
fi

Nothing earth shattering.

IPv6 Regex

Working with IPv6 addresses is a bit more complex. To compensate for the larger addresses size when representing IPv6 addresses in text, the RFC recommends a canonical textual representation with rules that allow for compression (called “zero folding”). Addresses represented in this compressed format are more difficult to validate with just one regex and the regex is much longer:

#!/bin/sh
WORD="[0-9A-Fa-f]{1,4}"
# flat address, no compressed words
FLAT="^${WORD}(:${WORD}){7}$"
# ::'s compressions excluding beginning and end edge cases
COMP2="^(${WORD}:){1,1}(:${WORD}){1,6}$"
COMP3="^(${WORD}:){1,2}(:${WORD}){1,5}$"
COMP4="^(${WORD}:){1,3}(:${WORD}){1,4}$"
COMP5="^(${WORD}:){1,4}(:${WORD}){1,3}$"
COMP6="^(${WORD}:){1,5}(:${WORD}){1,2}$"
COMP7="^(${WORD}:){1,6}(:${WORD}){1,1}$"
# trailing :: edge case, includes case of only :: (all 0's)
EDGE_TAIL="^((${WORD}:){1,7}|:):$"
# leading :: edge case
EDGE_LEAD="^:(:${WORD}){1,7}$"
is_ipv6 () {
    echo $1 | grep --silent "(${FLAT})|(${COMP2})|(${COMP3})|(${COMP4})|(${COMP5})|(${COMP6})|(${COMP7})|(${EDGE_TAIL})|(${EDGE_LEAD})"
    if [ $? -eq 0 ]; then
        return 1
    fi
    return 0
}

is_ipv6 $1
if [ $? -eq 1 ]; then
    exit 0
else
    echo "Invalid IPv6 address: $1" >&2
    exit 1
fi

Folks on the web have got it right too and I definitely took a queue from Vernon Mauery. I got a bit caught up in the differences between addresses from RFC4291 and the recommendations in RFC5952. The prior allows for zero folding of single 16-bit 0 fields while the latter discourages this. As the “robustness principle” dictates this validation script will identify addresses with zero folded single 16-bit 0 fields as valid but tools producing addresses should not.

I haven’t taken on any of the weirdness that are mixed hexadecimal and dot decimal notations … those will remain for the interested reader.

dnsmasq and racoon VPN

I’ve always used the standard dhcp3 server on my local networks (non-routable IPs). Never really knew of any other options and I didn’t look for any. As the few networks I manage have gotten larger I’ve wanted my DHCP server to be able to feed information into local DNS so I don’t have to maintain hosts files or remember IP addresses. I’ve heard horror stories about configuring BIND so I figured hooking up DHCP and BIND would be way too much work for my purposes.

After some digging I ran across dnsmasq [1]. It’s a DHCP server and a DNS proxy rolled into one. As it doles out DHCP leases it populates the DNS proxy with host names, just what I need. There are a good lot of howto’s out there for setting up dnsmasq so I won’t pollute the web with another one that’s likely not as good as the others. Frankly, dnsmasq can pretty much be configured with the information contained in its well documented example config file (provided as part of the Debian package).

What I will add to the inter-tubes is how I got dnsmasq to resolve names for VPN users connected to the racoon VPN that I’ve documented in a previous post [2] without interfering with other DNS configurations on the client. This requires a few modifications to the racoon server config and the client side up/down scripts. It also takes some resolvconf magic to finish the job.

Serving DNS info to VPN Clients

The configuration required to get racoon to send DNS information to clients as part of the mode_cfg is pretty straight forward.

    dns4 10.XXX.XXX.XXX;
    default_domain "vpn.example";

That’s it. The client side up script receives these config parameters in two new environment variables: INTERNAL_DNS4_LIST and DEFAULT_DOMAIN. The INTERNAL_DNS4_LIST reflects the fact that we can include the address of more than one DNS server. In this example we’ve only got one but we write our script such that it can handle the list.

In the up script we’ve got the DNS information now but what do we do with it? I’m no expert at building a resolv.conf files by hand and I really don’t want to be. We need a way to manage multiple DNS configurations at the same time such that when we need to resolve names for hosts on the VPN network they get routed to the DNS server configuration received from racoon. Other names we want resolved by whatever DNS configuration was in place when we brought up the VPN connection. The resolvconf program (yeah bad/confusing choice of names) almost does what we need.

Client Configuration with resolvconf

The man page for resolvconf is pretty straight forward but it leaves out one specific detail. By my reading of the man page I would think to call resolvconf as follows:

  echo -e "domain ${DEFAULT_DOMAIN}nnameserver ${DNS_IP}" | resolvconf -a ${INTERFACE}

Where INTERFACE would be the name of the interface we’re talking to the VPN through.

This doesn’t actually work though. After an hour of trying multiple configurations to see what I was doing wrong I thought to look at the script that the resolvconf package installed in my /etc/network/if-up.d directory. This script takes whatever DNS info was associated with the interface either statically in the interfaces file or dynamically over DHCP and feeds it into the resolvconf program. It does something funny though. The interface name used isn’t actually that of the interface. The script appends the address family to the interface name passed into resolvconf.

I tried using this convention for the VPN configuration scripts. I appended ‘.vpn’ to the interface name (very original I know) and this time the DNS info obtained over the VPN doesn’t stomp all over the existing DNS info (the configuration my laptop got from DHCP on the local network). The small addition to the racoon up script is as follows:

RESOLVCONF=$(which resolvconf)
INTERFACE=$(ip route get ${REMOTE_ADDR} 
    | grep --only-match 'dev[[:space:]][0-9a-zA-Z]*' | awk '{print $2}')
if [ -x ${RESOLVCONF} ]; then
    INPUT=""
    for DNS in ${INTERNAL_DNS4_LIST}
    do
        INPUT=${INPUT}$(echo "nameserver ${DNS}")
    done
    echo -n -e "domain ${DEFAULT_DOMAIN}n${INPUT}" 
        | resolvconf -a "${INTERFACE}.vpn" | logger -t "phaseone-up"
fi

This is a step in the right direction but it still doesn’t work exactly as we want.

The resolv.conf file generated by resolvconf after bringing up the VPN looks like this:

nameserver 192.XXX.XXX.XXX
nameserver 10.XXX.XXX.XXX
search home.example vpn.example

Here the 192.XXX.XXX.XXX DNS server was obtained by our network interface when it was brought up using DHCP. This is the DNS server on my home network. It knows the names of devices that have registered using DHCP and when searching for a hostname that’s not qualified the suffix appended is ‘home.example’. I leave off the top level suffix to prevent the proxy from forwarding bad search requests. The 10.XXX.XXX.XXX DNS server is the one that will resolve hosts on the VPN network. Again it knows the names of devices that have registered on the VPN network using DHCP and provides the search suffix of ‘vpn.example’.

Why This Doesn’t Work

Because the home DNS server is listed before the VPN DNS server it will be queried first. When asked for a host that exists on the VPN domain the query will first be sent to the DNS server on the home.example netowrk and the query will fail. The query will fall through to the next nameserver only in the case of a timeout or an error so the VPN DNS server will not be queried in this case and we can’t resolve names on the VPN network. If we switch their order manually we’ll be able to resolve names on the vpn.example network but attempts to resolve names on the home.example network will fail.

This situation is represented graphically here:

To make this more concrete, say we want to resolve the name for ‘bob’ (like if I were to run ‘ping bob’), a system on the home.example network. We’d expect the resolver to be smart enough to search through the two available DNS servers knowing their search domains. It could ask the vpn.example DNS server for ‘bob.vpn.example’ and if I didn’t find bob there it could then ask the DNS server on home.example for ‘bob.home.example’. If only the resolver functions in libc were this smart.
NOTE: we’d be in trouble if each network has a host named ‘bob’ but how to handle that situation is out of scope for this discussion.

For configurations that are relatively advanced we have to fall back on a DNS proxy like dnsmasq. Yes we’re already running dnsmasq as a DNS proxy on these two networks but the problem we’re running into is that the resolver on the client isn’t smart enough. The smarts we need are built into dnsmasq.

dnsmasq as a Client-side DNS Proxy

Installing dnsmasq on the client is painless. It’s already tied into the resolvconf system so its notified of changes to the reslover information but it preserves the behavior of the standard libc resolver described above. We can however statically configure dnsmasq to consult a particular DNS servers for a specific domain with one configuration line:

server=/domain/ip

For the network layout described we could add two lines to the dnsmasq.conf file to get the behavior we want:

server=/home.example/192.XXX.XXX.XXX
server=/vpn.example/10.XXX.XXX.XXX

Static configurations stink though (too easy) and with a little more work we can get the same effect with a short script:

#!/bin/sh                                                                       
# ip address from string to int                              
function inet_aton () {
    local count=3
    local int=0
    for num in $(echo $1 | sed -e 's/./ /g'); do
        let "int+=$num*256**$count"
        let "count-=1"
    done
    echo $int
}
pushd "/etc/resolvconf/run/interface/" > /dev/null
FILES=$(/lib/resolvconf/list-records | sed -e '/^lo.dnsmasq$/d')
for file in $FILES; do
    ns=$(cat $file | sed -n -e 's/^[[:space:]]*nameserver[[:space:]]+//p')
    PARAMS+="uint32:$(inet_aton $ns) "
    domain=$(cat $file | sed -n -e 's/^[[:space:]]*domain[[:space:]]+//p')
    PARAMS+="string:$domain "
done
dbus-send --system  --dest='uk.org.thekelleys.dnsmasq' 
    /uk/org/thekelleys/dnsmasq uk.org.thekelleys.SetServers $PARAMS
popd > /dev/null

For this script to make sense it’s important to know that when the resolvconf system is passed DNS information for a particular interface it makes a file with the name of the interface in /etc/resolvconf/run/interface/. This last script is placed in the directory /etc/resolvconf/update.d/. Each script in this directory is run every time the resolvconf information is changed. In the script we extract the nameserver and domain information from each file and send it to dnsmasq through the dnsmasq dbus interface (which must be enabled in dnsmasq.conf).

That’s it. Now each time we make a connection to the VPN the racoon client scripts send the VPN DNS info into resolvconf. resolvconf then runs its update.d scripts and the new script that we’ve provided takes this DNS information and sends it through to dnsmasq through a dbus interface. That was a lot of work, but now my VPN works the way I want it to. Well worth the effort IMHO.

I’m no dbus expert but I don’t really like the dnsmasq dbus interface. All functionality for manipulating the servers is packed into one function. As you can see from the above script it’s just “SetServers”. The interface would be much more effective and much easier to use if this one function were broken up into several, like an “AddServer”, “RemoveServer” etc. The full documentation for the dnsmasq dbus interface can be found here [3]. Proposing a few patches to fix this up would be a fun summer project 🙂

Racoon IPsec VPN on Debian Lenny

I’ve been wanting to set up a “pure” IPsec VPN using racoon for a while now. Part just for fun, part because I can. I spent a weekend on it once a while back, didn’t make much progress, got sick of trying to figure out cryptic racoon debug output and then gave up (more pressing stuff, you know how it is).

Anyways I ran into a situation where I NEED a VPN now. I manage a few systems that are facing the internet so they’re constantly under attack (mostly bots trying to brute force ssh all the time). Remote administration over ssh is pretty much all I need but I’d like to be able to keep a closer eye on the hardware through the “Integrated Lights-Out” system (they’re HP Proliant servers). These I don’t want facing the internet. Similarly I don’t want the configuration interfaces for my switch facing the public either.

So what I needed was a management network that I could connect to through a VPN gateway remotely (typically known as a “roadwarrior” setup). The ALIX system I’ve been blogging about in the recent past is what I’m using as the gateway / server. This post is a quick run down of the contortions I went through to get this working and why I didn’t get it working just how I want it 😦

Requirements

  • racoon only, no L2TP
  • rsasig authentication
  • as little hard coded network configuration as possible on the client

I thought the above was pretty ambitious. Configuring racoon is pretty complicated but after pouring over the man page for racoon.conf I found the mode_cfg section which specifies the network configuration for the server to send out to authenticated clients. A little more digging turned up a few examples, particurlarly useful were the netbsd howto and the howto forge racoon roadwarrior configuration.

Both of these give a working example of using the hybrid_rsa authentication with mode_cfg. This isn’t exactly what I wanted but it solves 2 of my 3 requirements above so it’s a great start. Next was adding my own server and client certificates to the configuration and making sure that both the certificate and the remote identifier were being verified. I didn’t want to keep having to type in a password when connecting to the VPN so I moved on to getting rsasig authentication working. Naturally at this point all hell broke loose.

It took me forever to figure it out, but it looks like the ipsec-tools version that ships with Lenny (0.7.1) doesn’t play nice with rsasig authentication and mode_cfg. The client and server are able to negotiate phase 1 without any troubles when the client never requests the configuration data from the server. I tried all sorts of configuration combinations hoping to find something that worked with no luck. Eventually I ran across an old and unanswered post to the ipsec-tools users mailing list from a few years back describing the same problem I’m having with the 0.7.0 version of racoon. Probably safe to assume that this behavior is what I’m running into on the version 0.7.1.

At this point my options were to upgrade to a later version and hope the bug was fixed or use hybrid auth. Guess which one I chose … hybrid auth ain’t so bad 🙂 Yeah typing in a password is a PITA but both client and server can still be configured to check each others certs and asn1dns identifiers in phase 1 so very little (if any) security is compromised. 2 out of 3 requirements isn’t bad. None of the desired functionality was lost but I do have to supply a password each time I connect to the VPN. Meh.

Configurations

Since the articles on howto forge and netbsd.org are so good I won’t bore you with a full description of my racoon configuration since it’s very similar. I will include them here for completeness and cover the parts where they differ.

The server.racoon.conf is very similar except for the credential verification for the client and some details in the mode_cfg. I’m using the split_network directive to send routing information to the client so we don’t have to hardcode any routes. The scripts on the client side had to be changed to accommodate this but it wasn’t that hard (I’ll get to this in a second). Also notice that I’m using all class C networks (/24 in CIDR) so no routes need to be specified on systems plugged into the management network directly.

In my client.racoon.conf the only difference is in the verification of the servers credentials (certs). I started out testing using certificates generated from my own root CA. When this was deployed I got certificates from cacert.org which is a great service.

The significant changes on the client side were in the phase one up and down networking scripts. I really didn’t like the scripts from the howto forge article (some of it didn’t seem necessary). They were a great starting place though but since I’m using the split_network I had to be able to properly handle dynamically creating and removing these routes. The final scripts can be found here: client.phaseone-up.sh and client.phaseone-down.sh

If you look closely you’ll notice that in the phaseone-down script I flush all SAs when taking down the VPN. Obviously this isn’t what we want: we want only to delete the relevant SAs. Unfortunately version 0.7.1 of the ipsec-tools package doesn’t play well with the way Linux manages SAs so the deleteall setkey function doesn’t work right. Flush is all we’re left with unless we’re gona parse the SADB output for the SPIs we need. This bug was reported on the ipsec-tools devel mailing list with a patch and it seemed well received so it’s likely in a later release. I’ll put off writing fancy bash script and just upgrade racoon soon.

Hope this is useful to everyone out there setting up a roadwarrior friendly racoon VPN. Leave me a comment if this was useful or if any of it’s unclear.

ThinkPad x61s UltraBay docking script

UPDATE: I’ve updated this script with a much better design because SELinux wouldn’t let me muck around with X’s tmp files. dock.sh.

Over a year ago I decided to pick up a new ThinkPad, this time the ultra portable x61s. My desktop was aging and I needed mobility more than anything else. I had always intended to get the X6 UltraBay so I could set it up on my desk but I never seemed to get around to it (probably on account of the phobia I developed while trying to get RedHat 9 to suspend and resume on my old T40).

Tonight, after getting my new (well new to me) UltraBay from ebay in the mail I took on setting up a script to run when docking and undocking the laptop. First off, I’m impressed with how well Linux works with the docking station “out of the box”. Even without udev rules to handle the dock/undock event the CD-RW/DVD combo drive hot plugs perfectly (I’m running a vanilla 2.6.31 kernel on Debian Lenny).

The udev rules and docking scripts for this set-up are well documented out there on the web but there were some details I had to assemble from a few different places. This post is mostly to collect the details in one place. If it’s useful to someone else out there even better.

First lets define what we’re trying to do: I’ve hooked up an external monitor to the UltraBase and I want to distribute my desktop across the external monitor and the LVDS display on the laptop when it’s docked. When it’s undocked the laptop should return to using only the LVDS display. We can script all that using a little bit of control logic and some very basic xrandr commands.

Let’s run through getting xrandr working first, then worry about when and how the script should be run. ThinkWiki has some great info for using xrandr to configure an external monitor. But when I ran some of these commands manually, nothing happened?

Turns out that when I first installed Debian on this laptop there wasn’t an external monitor attached (big surprise) so when X was configured it generated a configuration file that couldn’t handle the external monitor. This means xrandr could query properties from the connected monitor just fine, but any attempt to change the configuration did nothing. The command gave a successful status code but nothing happened.

The suggested fix is to attach the second monitor and reconfigure X. On Debian we’d expect this to be done through dpkg:

dpkg-reconfigure xserver-xorg

Which does nothing. Turns out you can tell X to generate a config file directly which does the trick this time:

sudo X -configure

I know nothing about X configuration files so I can’t say why this works but it does because now when we send X commands through xrandr it responds.

Start small by just turning on the external display:

xrandr --output LVDS --auto --output VGA --auto

Your displays may be named a bit differently which you can check using the query option (-q). The external monitor turned on but the screens overlapped in a funny way … progress.

The effect I want is to have my desktop extend across both screens so they’re side by side. xrandr does this without any hassle:

xrandr --output LVDS --mode 1024x768 --output VGA --mode 1024x768 --left-of LVDS

The command above makes the layout pretty obvious: both screens are 1024 by 768 and the VGA screen is positioned to the left of the built in LVDS. Some people want their external screen to have a higher resolution but it’s easy to change the configuration so I’m going for symmetry to start off. That and the larger the screens get the more RAM the video card borrows from the rest of the system.

This is the command we want to run when the laptop is docked, now the command when it’s undocked:

xrandr --output VGA --off

The two commands can be combined into a script which we can pass a parameter, maybe a 1 when the system is being docked, a 0 when it’s undocked. Throw in a case statement and then we can test it:

#!/bin/sh
case $1 in
    0)
        echo "undock" | logger -t $0
        xrandr --output VGA --off
        ;;
    1)
        echo "dock" | logger -t $0
        xrandr --output LVDS --mode 1024x768 --output VGA --mode 1024x768 --left-of LVDS
        ;;
    *)
        echo "unexpected input" | logger -t $0
        exit 1
        ;;
esac
exit 0

Make this file executable and put it in /etc/thinkpad, call it dock.sh

Now to get the system to run the script for us when the laptop is docked and undocked. Debian Lenny uses udev so this is as simple as adding the following in a file in /etc/udev/rules.d/

KERNEL=="dock.0", ATTR{docked}=="1", RUN+="/etc/thinkpad/dock.sh 1"
KERNEL=="dock.0", ATTR{docked}=="0", RUN+="/etc/thinkpad/dock.sh 0"

I named this file 55-thinkpad-local.rules based on someone’s reported success on the linux thinkpad mailing list. The order in which udev rules are run, why the order is important and how to write them is still a mystery to me and will remain that way for now.

Now we put the two together. The log messages that are sent to the syslog should be verified to be sure the script is running right because when the laptop is docked/undocked … the screen layout won’t change! Great right? We can get the error message by redirecting the output of the xrandr commands to syslog too like so:

xrandr --output LVDS --mode 1024x768 --output VGA --mode 1024x768 --left-of LVDS 2>&1 | logger -t $0

The error message tells us that the script “Can’t open display”. Wait, root (the script is run as roor) doesn’t have permission to open the display? This turns out to be some X magic that’s explained on the ThinkWiki Fn-F7 page. The important part is way down at the end of the script where root enumerates the X sessions and then one by one exports the server identifiers and the Xauthority cookie. After this root can change the display all it wants. We’ve gotta include this in the original script and while we’re at it throw in some extra stuff to make it pretty. The final script is here. Works pretty good for me, but YMMV.

Extracting table schema from database

As part of rehabilitating an old project I’ve had to go through a fairly sizable MySQL database and extract the table structure. I’m doing this for knowledge capture. Currently the schema is only known to the internals of the software that operates on the database, there’s no documentation to speak of (oh what I’d do to get an ER diagram and some notes from the original developers).

In the scheme of things this is a pretty small database (well two of them actually), with only a few dozen tables. Doing the extraction by hand wouldn’t be overly time consuming but it would be overly mind numbing. And as the adage goes: “anything worth doing once is worth scripting”, so I wrote up a quick shell script that will dump the CREATE TABLE statements for each table in a set of databases.

#!/bin/sh
#  extract table schema from some set of databases
#
DB_USER=you
DB_PASSWD=yourpasswd
PARAMS="--skip-column-names -u ${DB_USER} -p${DB_PASSWD}"
DB_SELECTOR='^(db0|db1|db2)$'
TABLE_SELECTOR='.*'

echo "SHOW DATABASES;" 
    | mysql ${PARAMS} 
    | grep ${DB_SELECTOR} 
    | while read DB; do
    echo "CONNECT ${DB}; SHOW TABLES;" 
        | grep "${TABLE_SELECTOR}" 
        | mysql ${PARAMS} 
        | while read TABLE; do
        echo "CONNECT ${DB}; SHOW CREATE TABLE ${TABLE}" 
            | mysql ${PARAMS} 
            | sed -e  "s/\\n/\n/g;s/^${TABLE}[t]//;s/AUTO_INCREMENT=[0-9]*//
g" > ${DB}-${TABLE}.sql;
    done;
done;

I’m a big fan of piping things into while read VAR statements. I’m sure some hate it since it’s a bit hard to read / see where the loop begins but it seems elegant to me and it’s very useful.

The script is basically three sql commands:

  • The first gets the names of all databases the user has access to. This could feasibly be a large number of databases so we add a “selector” to give us a regex we can use to narrow this set down.
  • The second SQL command gets the tables from the selected database. This is within a while loop so it is executed once for each database. This set of tables is also filtered through another regex.
  • Finally in the third SQL command we dump the create table statement for each table (within yet another while loop).

This gets us the CREATE TABLE statements but MySQL puts some stuff in here we don’t want:

  • It prefixes each CREATE TABLE statement with the table name for some reason
  • It also includes the AUTO_INCREMENT value which we don’t care about either
  • It also doesn’t put new line characters in the output but does put the string “n”.

We pipe this output through a pretty gnarly looking sed find/replace to clean it up. The output is then redirected to a file with a name formatted to contain both the name of database and the name of the table. That’s all there is to it. I’ve put up dummy values for some of the selectors as an example.

The script can be downloaded here.

NOTE: The grep and sed regexs can be pretty confusing because of some bash double quoting and escaping. Knowing the rules for bash escaping is essential … and not covered here 🙂

Cheers!