Verizon customer support is crap

RANT ALERT!

I’m writing this post as I’m waiting in a Verizon tech support phone queue. I’ve now waited through 4 different queues and spent 20 minutes convincing the first level tech that the reverse DNS PTR record I need in their DNS servers has nothing to do with DHCP. I’ve been on hold for about 2 hours now and if that wasn’t enough to drive me crazy, this stupid muzak is will.

As much as I want to I won’t use this space to tear on the Verizon techs for being stupid. In fact some of them are actually quite knowledgeable and I understand the need to have varying levels of technical expertise in such a large organization. What I will fill this space with is a testimate to the fact that Verizon specifically has their “customer service” infrastructure set up to deter people from asking for, or getting help. They’re content to spin you through an endless cycle of phone queues until you give up and go away.

How it all got started

I don’t send a whole lot of mail from my domain, just my personal stuff, but a month or so back I started to notice that email I’m sending to Gmail users was ending up in their spam folders. No one goes through their spam folders so in the end my friends weren’t getting my emails. A little digging shows that this is likely caused by my lack of and SPF text record in DNS.

That’s easy enough to populate myself through the GoDaddy servers, but for SPF to work correctly it requires reverse DNS to resolve the name of a host that’s authorized to send email fro your domain. Unfortunately this isn’t something I can set up myself since Verizon owns the IP of my mail server and controls the reverse DNS record (PTR).

So how do I get this record populated? I call Verizon tech support. They then tell me that I have to make the request in writing (email) which I did 2 days ago. Well today I get a very nondescript email which I’ll produce here in its entirety:

Hello,

I apologize, but Verizon is unable to process DNS requests at this time.

If you have any further questions regarding this particular issue, please
respond to this email leaving the ticket number in the subject line.

Thank you for using Verizon Business,

Erik Correll
Verizon Business
866-492-1074 option 2

This Means War

Thanks Erik, that’s about the most unhelpful response you could give me. So what you’re saying is: Computer Says No. Next time just tell me the truth which would read something like this: “I’m busy, go away for a while and try back later”. The odds of me responding to this email and waiting another 2 days is 0. I got on the phone this time determined to wait in as many phone queues as I have to till this is resolved.

Funny thing is the first knowledgable person I talk to (1 hour of waiting) told me I had to make the request over email. Nuts to that I already did. I send him off to get me to the right department and it turns out he has the wrong phone number.

Verizon plays this cute little trick where they change the phone numbers of their departments on a continual basis. This is to prevent people from saving the numbers of specific departments and calling there directly. It’s also intended from keeping awesome sites like DSLReports from publishing these numbers accurately. What a bunch of jerks right?

DSL Reports has a FAQ page specifically dedicated to getting reverse DNS records populated through Verizon and guess what, the phone number they publish is wrong now. It was accurate when the published the FAQ but Verizon changed it to keep people from calling that department directly.

This guy was actually very helpful and eventually he gets me through to the DNS guys. Again they tell me I have to make my request in writing. Yeah I already did that. Another half hour of waiting and this guy comes back on the line and tell me that they have a 30 day freeze on all DNS changes and he didn’t know about this till I asked! That’s the biggest WTF I’ve ever heard. He’s in their DNS department and he didn’t know about this freeze till I asked. Right well I’m getting my damn reverse DNS PTR if it kills me, or it costs me all of the minutes I have left on my cell plan for the month. That’s right this is actually costing me money.

So yeah this isn’t even resolved yet. I’m still on hold after 4 hours of waiting for Verizon to figure out what their DNS policy currently is. So yeah, fiber to the home in the US is crap. It’s owned by the biggest most disfunctional monopoly on the face of the earth. I pay a small fortune for fast fiber and a static IP and I get crap for service.

Verizon has pretty much done everything they can to deter me: a blanked freeze on DNS service to their business customers (me) with no explanation and 4 hours of running me around on the phone. I will get this set up if it’s the last thing I do and I’m so bored waiting on the phone that I’m using this time to rant. It feels good but ultimately it doesn’t do any good. The only thing that will fix this is breaking up these huge telco monopolies so that there’s competition in this market. The only way we (customers) can get the attention of Verizon is if we take our money elsewhere by switching to a different provider. Right now, for high speed fiber to the home we can’t do this.

Alright this rant isn’t helping my blood pressure. I’m almost on to hour 5 of this stupid phone music and I’m ready to explode.

UPDATE: 2010-05-20
Just to finish the story: I never got this issue resolved over the phone. At just about the hour 5 mark my call got dropped by the Verizon help center. It just clicked a few times and then the line went dead. I fell back to sending emails with stronger language (i.e. “this is completely unacceptable” blah blah blah) and after a few more days they finally populated DNS with my PTR entry. I never got an explanation as to why they refused in the first place since this is a service they’re supposed to provide to business customers.

Lesson learned: don’t waste your time trying to get through to Verizon tech support on the phone, just send them a hand full of increasingly angry emails.

xend fails silently

I’ve been giving myself a crash course in Xen recently. I’ve played with Xen in the past but it’s always annoyed me that the kernel for dom0 was super old (2.6.18). Debian has always been good with their support and the 2.6.26 kernel that ships with Lenny has worked well.

Now that Xen 4.0 is out and I’m getting ready to transition over to Squeeze, I wanted to go back and find where the upstream dom0 kernel was maintained. Turns out there’s a git tree hosted on kernel.org with a stable 2.6.32 and more bleeding edge kernels as well. Building both Xen 4 and the 2.6.32.10 kernel from git were pretty uneventful. I did however run into a problem getting xend to start.

There’s nothing worse than a program that fails without giving any good indication as to why. xend failed with this error message in /var/log/xen/xend-debug.log:

Error: (111, 'Connection refused')

That’s it. Not helpful at all. I started fishing through an strace but got lost in the output pretty quick. Luckily there was some help to be had in the #xen channel on freenode.

The folks on IRC told me it’s quite common for xend to fail quietly like this if there’s a needed xen kernel module that isn’t loaded. The usual suspects are xen-evtchn or xen-gntdev but it may be some other component that wasn’t compiled in. In my situation I had xen-gntdev built in but xen-evtchn was built as a module and hadn’t been loaded. A quick modprobe later and xend was up and running. I went one step further and recompiled this piece into the kernel directly.

Now I’m happily running Xen 4.0 with a 2.6.32 kernel on Debian Squeeze. Good times.

Raleigh Record: Mission Accomplished

It’s done. Putting the bike back together was super easy. Part of what made it so easy was that I put the components (breaks, derailers etc) back together as I cleaned them up. The brakes were already done so I just had to bolt them back on, the bottom bracket was a bit more difficult since I had to order new cups and cotter pins but the local bike shop made that easy enough.

Hardest part was installing the new head set. Pressing the cups was easy enough but getting the bottom race on the forks was a pain. The times I’ve fit a race on a set of forks it’s been pretty straight forward. This time the race just didn’t fit on the forks. Again, Steve at the bike shop stepped up with a grinder and “made it fit”. Then I just had to get a spacer for the headset, run new cables, and fit the seat.

The seat was a Christmas gift from my mom (thanks mom!) and is the nicest part on the bike. It’s a Brooks B17 and it’s super nice and super hard. Breaking this thing in is gonna take a while. I also ordered leather bar tape to match the seat and had no idea how expensive it is. Leather is nice but it’s super expensive.

Doing a little digging turns up the Raleigh Record is Raleigh’s entry level bike of the time. I read a few posts from people who paid $20 for a similar bike in perfect condition so restoring this was definitely a labor of love. Now I’ve got a $20 bike with a $200 seat. Oh the irony.

And now the pictures:

Running OP for the First Time

OP is a little wonky to run the first time. Not because it doesn’t work but because it doesn’t appear to. Sounds strange right? Well the first time I fired it up (by running the run-op2 script in the root of the source tree) it loaded up the web page of one of the students working on the project. His name is Shou Tang [1].

Unfortunately OP loads most of his page, sometimes the whole thing, and then seems to get lost. Attempts to load any other page will fail. The little icon in the tab which indicates that it’s “thinking” continues to spin but nothing happens. If you fire up tcpdump and watch for http traffic you’ll see that OP is making web requests and getting responses but it doesn’t render the page at all.

Unfortunately there isn’t a whole lot of debug output. Since I’m not very familiar with the architecture yet there may be some debug that I just don’t know where to find. There is a warning that gets dumpted to the console indicating that a clientClosed signal is being ignored this warning is generated by a lot of pages, not just Shou’s page. I did get a few weird errors from X but I couldn’t seem to reproduce these reliably.

So to get OP working you need to fire it up using the run-op2 script and let it get all messed up by Shou’s page. Then pull down the Menu button that’s in the upper right side of the screen. Go into the Edit tab and select Preferences. In here change the default home page to something that won’t muck up the browser (I chose Google). Then restart the browser. This time you’ll start out on a different home page and hopefully this one won’t screw up OP.

I’m gonna hold off on speculating as to why Shou’s page messes up OP. Fixing this bug would be nice but it’s not really my focus. My next post will detail how OP starts up and hopefully some architectural details (which parts talk to each ether etc). Documenting this will be the first step towards laying out what a Type Enforcement policy (a la SELinux) for OP should look like.

[1] http://www.cs.uiuc.edu/homes/stang6/