Author: flihp

I'm me.

change jar

~ flihp ~ Leave a comment

I’ve been a slacker this past month and haven’t posted much technical stuff. Once class ended and work picked up for the summer I pretty much fell off the map. I’m working on a post now to describe the work I did over this past semester and hopefully where it will go over these next few months.

But for now I’m falling back on some filler. That’s right, completely useless information follows: my change jar is full!

Now it’s time to roll it up. Any guesses on how much $$ is in there? There’s lots of quarters so I’m guessing it’s gonna be around $400.

Volume Cutter

~ flihp ~ Leave a comment

It’s been a busy month and I’ve neglected putting up a few things I’d deem “blog worthy”. First up is the new bike I’m building. Restoring the Raleigh was so much fun I had to start in on another one.

So far I’m just spec’ing out parts. I settled on the Cutter frame by Volume [1]. They’re a BMX company that’s started to dabble in fixie frames. This is their 2009 model the paint is sick. It’s sort of a toothpaste green and it glows in the dark.

I picked up the matching “fu manchu” fork in white. I couldn’t get the fork in the same glow-in-the-dark paint since it’s last years model. I like keeping the fork and frame different colors and I’ll be sticking to a white contrast color for the wheels and bars too.

I’m not sure I like the stem and bars. They’re mountain bike parts and they’re pretty beefy. I like the rise on the stem but I don’t like how thick the bars are and they’ve got a bit of a sweep to them too. Details.

The best thing about this frame is how much like a BMX it is. It’s got an integrated headset that works with campy spec bearings. I got a set of Odyssey bearings off ebay on the cheap and they slip right on. Odyssey started putting a cut in the crown race that makes fitting it on the forks super easy. This makes it so you don’t have to pound it on.

The Volume logo on the head tube isn’t a sticker or paint. It’s actually cut out of the head tube. Cool idea, but I’m taking bets on what the first piece of debris that falls down in there will be.

So far the only down side to this frame is the weight. It’s built like a BMX frame so it’s heavier than other road frames. Other than that I love it. That’s pretty much all I’ve had the time to put together. Hopefully this weekend I’ll order the wheels, then start looking into parts for the drivetrain.

References:
[1] – http://www.volumebikes.com/

Verizon customer support is crap

~ flihp ~ 2 Comments

RANT ALERT!

I’m writing this post as I’m waiting in a Verizon tech support phone queue. I’ve now waited through 4 different queues and spent 20 minutes convincing the first level tech that the reverse DNS PTR record I need in their DNS servers has nothing to do with DHCP. I’ve been on hold for about 2 hours now and if that wasn’t enough to drive me crazy, this stupid muzak is will.

As much as I want to I won’t use this space to tear on the Verizon techs for being stupid. In fact some of them are actually quite knowledgeable and I understand the need to have varying levels of technical expertise in such a large organization. What I will fill this space with is a testimate to the fact that Verizon specifically has their “customer service” infrastructure set up to deter people from asking for, or getting help. They’re content to spin you through an endless cycle of phone queues until you give up and go away.

How it all got started

I don’t send a whole lot of mail from my domain, just my personal stuff, but a month or so back I started to notice that email I’m sending to Gmail users was ending up in their spam folders. No one goes through their spam folders so in the end my friends weren’t getting my emails. A little digging shows that this is likely caused by my lack of and SPF text record in DNS.

That’s easy enough to populate myself through the GoDaddy servers, but for SPF to work correctly it requires reverse DNS to resolve the name of a host that’s authorized to send email fro your domain. Unfortunately this isn’t something I can set up myself since Verizon owns the IP of my mail server and controls the reverse DNS record (PTR).

So how do I get this record populated? I call Verizon tech support. They then tell me that I have to make the request in writing (email) which I did 2 days ago. Well today I get a very nondescript email which I’ll produce here in its entirety:

Hello,

I apologize, but Verizon is unable to process DNS requests at this time.

If you have any further questions regarding this particular issue, please
respond to this email leaving the ticket number in the subject line.

Thank you for using Verizon Business,

Erik Correll
Verizon Business
866-492-1074 option 2

This Means War

Thanks Erik, that’s about the most unhelpful response you could give me. So what you’re saying is: Computer Says No. Next time just tell me the truth which would read something like this: “I’m busy, go away for a while and try back later”. The odds of me responding to this email and waiting another 2 days is 0. I got on the phone this time determined to wait in as many phone queues as I have to till this is resolved.

Funny thing is the first knowledgable person I talk to (1 hour of waiting) told me I had to make the request over email. Nuts to that I already did. I send him off to get me to the right department and it turns out he has the wrong phone number.

Verizon plays this cute little trick where they change the phone numbers of their departments on a continual basis. This is to prevent people from saving the numbers of specific departments and calling there directly. It’s also intended from keeping awesome sites like DSLReports from publishing these numbers accurately. What a bunch of jerks right?

DSL Reports has a FAQ page specifically dedicated to getting reverse DNS records populated through Verizon and guess what, the phone number they publish is wrong now. It was accurate when the published the FAQ but Verizon changed it to keep people from calling that department directly.

This guy was actually very helpful and eventually he gets me through to the DNS guys. Again they tell me I have to make my request in writing. Yeah I already did that. Another half hour of waiting and this guy comes back on the line and tell me that they have a 30 day freeze on all DNS changes and he didn’t know about this till I asked! That’s the biggest WTF I’ve ever heard. He’s in their DNS department and he didn’t know about this freeze till I asked. Right well I’m getting my damn reverse DNS PTR if it kills me, or it costs me all of the minutes I have left on my cell plan for the month. That’s right this is actually costing me money.

So yeah this isn’t even resolved yet. I’m still on hold after 4 hours of waiting for Verizon to figure out what their DNS policy currently is. So yeah, fiber to the home in the US is crap. It’s owned by the biggest most disfunctional monopoly on the face of the earth. I pay a small fortune for fast fiber and a static IP and I get crap for service.

Verizon has pretty much done everything they can to deter me: a blanked freeze on DNS service to their business customers (me) with no explanation and 4 hours of running me around on the phone. I will get this set up if it’s the last thing I do and I’m so bored waiting on the phone that I’m using this time to rant. It feels good but ultimately it doesn’t do any good. The only thing that will fix this is breaking up these huge telco monopolies so that there’s competition in this market. The only way we (customers) can get the attention of Verizon is if we take our money elsewhere by switching to a different provider. Right now, for high speed fiber to the home we can’t do this.

Alright this rant isn’t helping my blood pressure. I’m almost on to hour 5 of this stupid phone music and I’m ready to explode.

UPDATE: 2010-05-20
Just to finish the story: I never got this issue resolved over the phone. At just about the hour 5 mark my call got dropped by the Verizon help center. It just clicked a few times and then the line went dead. I fell back to sending emails with stronger language (i.e. “this is completely unacceptable” blah blah blah) and after a few more days they finally populated DNS with my PTR entry. I never got an explanation as to why they refused in the first place since this is a service they’re supposed to provide to business customers.

Lesson learned: don’t waste your time trying to get through to Verizon tech support on the phone, just send them a hand full of increasingly angry emails.

xend fails silently

~ flihp ~ 2 Comments

I’ve been giving myself a crash course in Xen recently. I’ve played with Xen in the past but it’s always annoyed me that the kernel for dom0 was super old (2.6.18). Debian has always been good with their support and the 2.6.26 kernel that ships with Lenny has worked well.

Now that Xen 4.0 is out and I’m getting ready to transition over to Squeeze, I wanted to go back and find where the upstream dom0 kernel was maintained. Turns out there’s a git tree hosted on kernel.org with a stable 2.6.32 and more bleeding edge kernels as well. Building both Xen 4 and the 2.6.32.10 kernel from git were pretty uneventful. I did however run into a problem getting xend to start.

There’s nothing worse than a program that fails without giving any good indication as to why. xend failed with this error message in /var/log/xen/xend-debug.log:

Error: (111, 'Connection refused')

That’s it. Not helpful at all. I started fishing through an strace but got lost in the output pretty quick. Luckily there was some help to be had in the #xen channel on freenode.

The folks on IRC told me it’s quite common for xend to fail quietly like this if there’s a needed xen kernel module that isn’t loaded. The usual suspects are xen-evtchn or xen-gntdev but it may be some other component that wasn’t compiled in. In my situation I had xen-gntdev built in but xen-evtchn was built as a module and hadn’t been loaded. A quick modprobe later and xend was up and running. I went one step further and recompiled this piece into the kernel directly.

Now I’m happily running Xen 4.0 with a 2.6.32 kernel on Debian Squeeze. Good times.

Raleigh Record: Mission Accomplished

~ flihp ~ 3 Comments

It’s done. Putting the bike back together was super easy. Part of what made it so easy was that I put the components (breaks, derailers etc) back together as I cleaned them up. The brakes were already done so I just had to bolt them back on, the bottom bracket was a bit more difficult since I had to order new cups and cotter pins but the local bike shop made that easy enough.

Hardest part was installing the new head set. Pressing the cups was easy enough but getting the bottom race on the forks was a pain. The times I’ve fit a race on a set of forks it’s been pretty straight forward. This time the race just didn’t fit on the forks. Again, Steve at the bike shop stepped up with a grinder and “made it fit”. Then I just had to get a spacer for the headset, run new cables, and fit the seat.

The seat was a Christmas gift from my mom (thanks mom!) and is the nicest part on the bike. It’s a Brooks B17 and it’s super nice and super hard. Breaking this thing in is gonna take a while. I also ordered leather bar tape to match the seat and had no idea how expensive it is. Leather is nice but it’s super expensive.

Doing a little digging turns up the Raleigh Record is Raleigh’s entry level bike of the time. I read a few posts from people who paid $20 for a similar bike in perfect condition so restoring this was definitely a labor of love. Now I’ve got a $20 bike with a $200 seat. Oh the irony.

And now the pictures:

Running OP for the First Time

~ flihp ~ Leave a comment

OP is a little wonky to run the first time. Not because it doesn’t work but because it doesn’t appear to. Sounds strange right? Well the first time I fired it up (by running the run-op2 script in the root of the source tree) it loaded up the web page of one of the students working on the project. His name is Shou Tang [1].

Unfortunately OP loads most of his page, sometimes the whole thing, and then seems to get lost. Attempts to load any other page will fail. The little icon in the tab which indicates that it’s “thinking” continues to spin but nothing happens. If you fire up tcpdump and watch for http traffic you’ll see that OP is making web requests and getting responses but it doesn’t render the page at all.

Unfortunately there isn’t a whole lot of debug output. Since I’m not very familiar with the architecture yet there may be some debug that I just don’t know where to find. There is a warning that gets dumpted to the console indicating that a clientClosed signal is being ignored this warning is generated by a lot of pages, not just Shou’s page. I did get a few weird errors from X but I couldn’t seem to reproduce these reliably.

So to get OP working you need to fire it up using the run-op2 script and let it get all messed up by Shou’s page. Then pull down the Menu button that’s in the upper right side of the screen. Go into the Edit tab and select Preferences. In here change the default home page to something that won’t muck up the browser (I chose Google). Then restart the browser. This time you’ll start out on a different home page and hopefully this one won’t screw up OP.

I’m gonna hold off on speculating as to why Shou’s page messes up OP. Fixing this bug would be nice but it’s not really my focus. My next post will detail how OP starts up and hopefully some architectural details (which parts talk to each ether etc). Documenting this will be the first step towards laying out what a Type Enforcement policy (a la SELinux) for OP should look like.

[1] http://www.cs.uiuc.edu/homes/stang6/

OP browser install

~ flihp ~ Leave a comment

Building / installing “research” software is always fun. OP was better than most as far as the building goes. There isn’t a way to install it (at least not through the build system) so we’ll leave that part out. For posterity the code I’m using is a tarball they put up on google code back in October [1]. I had hoped to use the svn tree that they advertise [2] but it’s just an empty directory, no code.

Their directions are pretty good. I started from their google code wiki page that has directions for doing the install on Ubuntu. I did the install on a Debian Squeeze system that I’m running as a VM on my laptop. OP uses the WebKit rendering engine and those of you familiar with building WebKit already know how long it’s gona take me to build this in a wimpy little VM 🙂

Packages

The first time running OPs build script it will fail. There’s a bunch of development software you’ll need that’s not part of a default Squeeze install. Save yourself some time and just apt-get install these packages:

apt-get install gcc g++ flex bison gperf qt4-qmake libqt4-dev libsqlite3-dev libphonon-dev libxext-dev x11proto-xext-dev libfontconfig1-dev

Some of these libraries like libphonon-dev and libxext-dev were discovered as dependencies through trial and error. I mean to say I ran the build script, it errored out with some cryptic error like a missing header file like SomethingPhonon.h and then I apt-cache searched for a development package with the keywords phonon and dev and found the right one. Trial and error is pretty time consuming when you’re compiling this on a very low powered machine. Some additional packages may be pulled in as dependencies but the above list should be enough to get you what you need. If you run into any problems recreating this let me know in the comments.

OP recommends that you install Qt directly from Nokia but everything built fine for me using the Qt4 shipped with Squeeze. There are webkit and libqt4-webkit packages on squeeze and I tried these first. I’m pretty sure the libqt4-webkit package is missing some headers that OP needs since the build failed looking for headers that are supplied in the Qt bindings from the WebKit source tree. Nothing’s perfect, just use the WebKit source and the Qt4 bindings that comes with it.

Building WebKit

This is the part where I start complaining about building WebKit. Not just how long it takes (that’s my laptops fault) but the crazy build system. I guess I’ve been spoiled by all the great open source packages out there that build with the standard ./configure && make && sudo make install. Webkit ships with an autobuild.sh which will bootstrap the standard gnu autotools infrastructure but it will only build the WebKit core, it won’t build the Qt bindings we need for OP.

OP goes above and beyond in that they ship a script that downloads the WebKit code from the “nightly build” that OP was developed against [3]). It applies a set of patches too.

You can build WebKit through the script supplied by OP or you can do it yourself. If you chose the latter all you need to do is:

wget http://builds.nightly.webkit.org/files/trunk/src/WebKit-r48592.tar.bz2
tar jxvf WebKit-r48592.tar.bz2
mv WebKit-r48592 web-app/WebKit
cd WebKit; cat ../webkit_patches/*r48592.diff | patch -p0; cd ../
./web-app/WebKit/WebKitTools/Scripts/build.sh --qt --release

That’s downloading the right nightly build, extracting it, renaming the directory (OP has this path hard coded in their scripts), patching it and running the build script. I did this manually because the build kept failing and I wanted to narrow down the problem. Even with all of the right libraries installed I was getting a strange error from g++ indicating that there was a bug in the compiler itself:

g++: Internal error: Killed (program cc1plus)
Please submit a full bug report.
See <file:///usr/share/doc/gcc-4.4/README.Bugs> for instructions.
make[1]: *** [obj/release/FrameLoader.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory `/home/myuname/opbrowser-release-2009_09_30/webapp/WebKit/WebKitBuild/Release/WebCore'
make: *** [sub-WebCore-make_default-ordered] Error 2

Google for this error real quick and usually the problem is the machine that’s doing the compile running out of RAM. There are some great mailing list posts with people compiling glibc on a system with 32MB of ram with 128MB swap space. That makes my VM look like a super computer (512MB of RAM and a 1GB swap disk). My first reaction was then to think that there was no way I was running out of RAM.

More swap!!

So how to test this? Run the build again and this time run in parallel the command free -m -s 2. This will poll your RAM and swap usage every 2 seconds printing some info to the console. Sure enough the build was using up all of my RAM and swap which is pretty ridiculous IMHO.

So just throw more ram at it right? Getting KVM to give this VM more RAM take a restart so we get on that giving it 1GB of RAM (double what it had previously) and leave the 1GB of swap alone. FAIL, free still shows us running out of both RAM and swap.

OK no messing around this time. On my host system I allocated a 5Gb KVM logical volume and passed this to the VM as an additional hard disk (another restart of the VM). I then dropped the old swap space and set this 5Gb disk as swap. This turned out to be enough. Watching free showed my swap usage going well over 2Gb … jeez that greedy. Something in this script is assuming that my system is pretty beefy.

One final problem I ran into was a significant number of undefined references turning up in the final linking. This is from the build failing so many times previously. Typically you’d hope the build system would rebuild anything that fails but that’s not the case here. In fact even if you run the build script with the --clean switch it doesn’t clean the enough to remove broken object files. I had to manually delete the WebKitBuild directory which is under the WebKit root and rebuild WebKit one last time. You’ll see this message when you’re done:

===========================================================
 WebKit is now built (2h:45m:25s). 
 To run QtLauncher with this newly-built code, use the
 "./run-launcher" script.
===========================================================

That’s right, almost 3 hours to build this beast.

Building OP

In comparison to WebKit, building OP was a breeze. The only additional libraries were a few from boost [4]. These are as follows.

apt-get install ant sun-java6-jdk libboost-dev libboost-regex-dev

Installing libboost-regex-dev will pull in a bunch of boost dev packages one of which is libboost-dev. I’ve included libboost-dev in the list above just for completeness.

I’m pretty sure the OpenJDK java packages would work but since we’re trying to minimize the possible problems we may run into I just grabbed the “non-free” sun packages. That way if I end up having to get in touch with the guys that wrote OP with a question / problem they won’t have the opportunity to say “we don’t support the OpenJDK packages, make sure you’re using the genuine Sun (Oracle?) Java”. If anyone gives this a go with the OpenJDK packages let me know how it turns out in the comments.

Once you’re done accepting the licensing agreement ::sigh:: run the build script and OP should be good to go. OP ships with a build script in its root named build.sh. Agian this makes some assumptions about your system since it passes make the -j4 flag. This is generally the option you’d pass to make if you’ve got 2 CPUs. Since my VM only has one I went through and removed it:

cat build.sh | sed 's/-j4//' > build-single.sh

Then run it and you should be good to go.

I couldn’t figure out how to actually install WebKit once it was built. OP takes this on by setting LD_LIBRARY_PATH and DYLD_LIBRARY_PATH environment variable in its launch script to contain the WebKit release library directory: web-app/WebKit/WebKitBuild/Release/lib. There is also a hard coded reference to that path in some of OPs Makefiles.

This can cause a problem if you don’t pass the WebKit build script the --release flag (like maybe you built it with --debug instead). OP won’t build right in this case. It will fail complaining about a bunch of undefined references. If you do this by mistake and you don’t want to rebuild WebKit (because it takes around 3 hours) you can just use a soft link.

So now it’s built. This post is long enough so I’ll comment on running OP next time.

[1] http://code.google.com/p/op-web-browser/downloads/list
[2] http://code.google.com/p/op-web-browser/source/checkout
[3] http://builds.nightly.webkit.org/files/trunk/src/WebKit-r48592.tar.bz2
[4] http://www.boost.org/

Raleigh Update #5

~ flihp ~ Leave a comment

This is the turning point. The derailers were the last two pieces of this bike that had to be cleaned up / repaired. Mechanically they’re pretty much the most complicated parts of the bike. Modern derailers have a lot of plastic on them but these ones are all steel. This makes them super heavy but means as long as they’re not bent they’ll last forever.

So here are some before pics. The black stuff all over them is a mixture of old grease and dirt. This stuff’s like cement and to get it off I used a pretty serious degreaser, some scouring pads and a steel brush to clean it out of the threaded parts and some of the hard to reach places.

Here are the before pics:

As always the bolts were a real pain to get out. I managed to get everything apart except for the bolt that holds the cable from the shifter. Luckily the bike shop had an extra kicking around that they just gave me. Gotta love those random boxes of spare parts.

The hardest part was getting it back together, not because I couldn’t remember how to though … but it did take some experimentation. It was hard because the spring that’s hiding under that black bushing is super strong. After breaking the whole thing down and cleaning it I had to re-pack the spring and use a screwdriver to leaver it into place so I could slip the bolt back in. I stabbed myself with the screwdriver a few times so that was awesome.

I really wish I had taken pictures while it was apart but I was so excited to start putting parts back on the bike that I completely forgot. I did take a few after shots though:

The rear derailer was in even worse shape. Again, not bent but completely caked with the dirt / grease mixture. This one has more moving parts: it has two wheels (with unsealed bearings) that guide the chain this time. So this time the whole thing’s caked with crap AND it has two guide wheels with bearings that are packed in cement instead of grease. Needless to say they don’t spin. Here are the before shots:


Again this one was a just a disassemble, clean, grease, reassemble. The cleaning this time was pretty intense. Had to team up the degreaser with a small finishing nail to scrape the crud (that’s a technical term) out of the guide wheels and a few other tight places. I really should have taken some shots when this thing was apart. It was pretty impressive.

Either way the story isn’t an interesting one: I cleaned it, got new bearings for the guide wheels and packed them with grease. Now it looks awesome and this thing is one step closer to being done. In fact these were the last two parts that needed cleaning. I’m waiting on new cups, races and bearings for the bottom bracket and then this thing gets put back together. Now the after shots:

CIS791: OP browser based project statement

~ flihp ~ Leave a comment

As part of the web security class I’m taking this semester we’re required to put together a project in the last 6-ish weeks of class. The intent is to get us familiar with doing research and formulating a project based on our research. This project is a short one, really I don’t expect to do much more than scratch the surface of a project and show how cool it could be given enough time. Gotta have something interesting to talk about by the end of the semester though but little more. What I don’t want it to be though is a boring project that evaluates some platform or technology.

After doing a bunch of reading about “safe” subsets of Java and JavaScript (which has decidedly no relation to Java) I came to the conclusion that JavaScript is a mess. I love the idea of object capability languages but as close to a safe subset of JavaScript as some have got, I’m pretty sure trying to fix JavaScript isn’t for me.

the OP web browser

After reading the OP web browser paper when it was published back in 2008 I started looking for a reason to play around with it. It’s much more in line with my interests (software architecture as well as security). I especially like the idea of structuring an application using well established architectural concepts from OS design.

Separation is a huge deal in the OS (think separate process address spaces) and in the application space its making a come back. Breaking the browser up into smaller components with well defined interfaces and communication semantics is a great idea. From the security perspective it keeps browser plugins / components from stomping all over each other when one gets compromised. It also is an excellent way to exploit multi core systems. I always get super pissed when my flash player pins one of my CPUs to 100% and the whole browser gets dragged down with it while the other CPU is sitting idle.

potential contribution of this work

What I’m interested in is, of course learning some of the insides of this browser. But specifically I’m interested in the code that is interposed between different components of the browser (aptly named the kernel) and how much like a reference monitor it is or can be. Also the range of security policies it does / could enforce would be very interesting to discuss.

This latter point may actually require some work as it’s likely that OP could enforce any number of policies but that may take some heavy lifting to generalize the enforcement logic (this is pure speculation at this point). This would start looking like the LSM work from the Linux kernel.

When we start talking about policies, the granularity with which policies can be specified becomes important. Subjects and objects in operating systems are well understood for the most part. In a web browser that’s not so clear. The obvious things that come to mind are plugins (including instances of the java script engine) as subjects and components of the DOM and passive user data as objects (cookies, history, saved passwords etc).

Being the SELinux fanboy that I am I’m pretty convinced that a lot of the enforcement can be done in a user space object manager. This gives us the policy language (type enforcement) and policy enforcement point (the Linux kernel) for free but leaves the details of object definition anbd labeling us. It also does nothing for us as far as verifying the expected properties of our reference monitor (complete mediation, tamperproof and analysis for correctness). The design of OP itself will likely be a big help in verifying these properties, but this isn’t something I plan to spend much time on.

Conclusion

So there isn’t much to conclude yet. This is a basic statement of the project I’m undertaking for the rest of this semester, a jumping off point. Hopefully it won’t be too painful but just getting OP to install is a non-trivial task (I’m currently waiting for webkit to compile). I’ve already ran into some quirks in their build system which were pretty easily fixed but there isn’t a mailing list for the project or anything so I’m trying to track down a way to communicate with the project owners beyond sending emails to their personal accounts. We’ll see how receptive they are to suggestions soon enough.

Raleigh update #4

~ flihp ~ 4 Comments

Drilling out cotter pins is a pain in the ass. I tried real hard to pound these things out but I ended up doing more harm than good. I tried pounding them out the good old fashioned way but I ended up bended the threaded end one one side and putting a good ding in one of the crank arms. Yeah that last one really pissed me off.

So I broke out the drill and the Dremel tool and went to town. I’ve never done this before, so half of this will be me recalling what I did wrong. The other half will be how I managed to get it right on the second try.

On the first pin I took the Dremel, cut off the threads and started drilling from the narrow end. I started out with a relatively small drill bit mostly because I was afraid to use a bit that was too big I figured this might damage the crank arm or the spindle. This was pretty naive. The spindle is hard as nails, well actually harder. It’s harder than drill bits either way. I think you’d be pretty hard pressed to do damage to the crank arms too but I didn’t get a chance to try.

After drilling my pilot I tried opening it up further with a 3/4″ bit. It kept grabbing the sides and eventually it broke the bit. Not only was I drilling with the wrong sized bits I was drilling the wrong end. After drilling the threaded end I tried again to pound it out. But I then realized that the place I wanted to hit with the hammer was gone, I just drilled it out. Crap. Using a punch I could get enough of a foot hold on the sides to give it a good bang but not enough.

I then took the drill to the other side and made the same mistakes as before (broke another bit). But this time I realized I could get a larger drill bit through. That’s because … well that’s just how cotter pins work. Long story short, drill from the wide end with the biggest bit you can manage (mine was 5/8″). Stay aways from the sides but don’t worry about hitting the spindle. If you do eventually hit the spindle it will guide you further in (remember it’s a lot harder than the drill bit) but don’t go so far you bind the bit between the spindle and the crank arm. After this it was short work banging the pin out once I could find a spot to seat the punch

Notice the hole in the side of the cotter. This is from the drill bit making contact with the spindle. It sounds kinda squeaky when it does but it’s no big deal.

On the second crank arm I didn’t make the same mistake. I went right in with the 5/8″ bit from the wide end. Drilled it as far as I thought I could get away with and then banged the pin out from the threaded end. Short and simple. This goes with out saying but you need a solid place to drill from. Ideally you’d have the whole bottom bracket in a vice but I found it enough to get the frame up on my work bench with the bracket pinned down on a few blocks of wood. I held it down with a C clamp when necessary.

Once the pins are out, the crank arms come right out. To get the rest of the bottom bracket apart it’s just a matter of getting the end caps off. These things are super hard to get a good grip on with your wrench. One of mine was very stubborn and I had to enlist the help of Steve down at Mello Velo.

Best part of getting the bottom bracket apart was finding the crap that was inside of it. I knew something was up since the cranks would barely turn. I guess the bird seed in there was getting in the way. Yeah bird seed. I’m guessing the squirrels must have been hiding food in the frame when it was doing time on a back porch. They could have been dropping this stuff down the seat tube I guess.

Even the spindle had seed stuck to it.

Then it was back to cleaning stuff up, this time the crank arms. They came clean enough but they’re pitted up in spots.

This bottom bracket wasn’t sealed. Just bearings, cups and the spindle to hold them in place. I don’t think you can even replace these things and it’s not worth trying. Mello Velo is trying to track down some sealed bearings that will fit the frame. Hopefully I’ll be able to find a spindle that will work with the new bearings and the old cranks. I don’t want to have to spring for a new set of cranks after going through all of this trouble to keep this bike as original as possible.

More to come.