Category: Chatter

For an unrelated project, I learned how to make WordPress permalinks. I made that change on the blog this morning. My testing shows that they work, and that incoming links are redirected correctly.

If you should see a problem, please drop me a note.

The folks at undeadly.org have started posting “how I discovered OpenBSD” stories. This isn’t a story of how I discovered OpenBSD, but rather why I like it. Before you ask, I don’t have similar stories about any other operating system, not even any other BSDs. I was guided to FreeBSD in 1995, and I discovered NetBSD on my own shortly after. (An earlier version of this was previously published in a small promo pamphlet handed out at a tech conference years ago.)

Back around 2000, my employer’s main business was designing Web applications, but once those applications were built our clients would turn around and ask “Where should we host this?” That’s where I came in, building and running a small but professional-grade data center for custom applications.

As with any new business, our hosting operation had to make the most of existing resources. Hardware was strictly limited to cast-off hardware from the web developers, and software had to be free. The only major expense was a big-name commercial firewall, purchased for marketing reasons rather than technical ones. With a whole mess of open-source software, we built a reliable network management system that provided the clients with a more insight into their equipment than their in-house people could offer. The clients paid for their own hardware, and so had fancy high-end rackmount servers with their chosen applications, platforms, and operating systems. As the business grew we upgraded the hardware – disk drives less than five years old are nice – but saw no need to replace the software.

One Monday morning, a customer that had expected to use very little bandwidth found that they had sufficient requests to devour twice the bandwidth we had for the entire datacenter. This affected every customer. If your $9.95/month web page is slow you have little to complain about, but if your multiple-thousands-of-dollars-a-month Web application is slow you pick up the phone and scream until the problem stops.

To make matters worse, my grandmother had died only a couple days before. Visitation was on Tuesday, the funeral Wednesday morning. I handed the problem to a minion and said “Here, do something about this.” I knew bandwidth could be managed at many points: the Web servers themselves, the load balancer in front of them, the commercial firewall, and even the router all claimed to have traffic management capacity.

Tuesday after visitation I found my cellphone full of messages. The version of Internet Information Server could manage bandwidth — in eight megabyte increments, and only if the content was static HTML and JPEG files. With several Web servers behind the load balancer, that fell somewhere between useless and laughable. The load balancer did support traffic shaping, if we bought the new feature set. If we plopped down a credit card number, we could have it installed by next Sunday. Our big-name commercial firewall also had traffic shaping features available, if we upgraded our service level and paid an additional (and quite hefty) fee for the feature set. That left the router, which I had previously investigated and found would support traffic shaping with only an IOS upgrade.

I was on the phone until midnight Tuesday night, making arrangements to do an emergency OS upgrade on the router on Wednesday night. I had planned to go to the funeral Wednesday morning, give the eulogy, go home and take a nap, and arrive at work at midnight ready to rock. The funeral was more dramatic than I had expected and I showed up at work at midnight sleepless, bleary-eyed, and upright only courtesy of the twin blessings of caffeine and adrenaline. In my email, I found a note that several big clients had threatened to leave unless the problem were resolved Thursday morning. If I hadn’t already been stressed out, the prospect of choosing a minion to lay off would have done the trick. (Before any of those minions start to think I care about them personally: I work hard training minions, and swinging the Club of Correction makes my arms sore. Eventually. I don’t like to replace them.)

Still, only a simple router flash upgrade and some basic configuration stood between me and relief. What could possibly go wrong?

The upgrade went smoothly, but the router behaved oddly when I enabled traffic shaping. Over the next few hours, I discovered that the router didn’t have enough memory to simultaneously support all of our BGP feeds and the traffic shaping functionality. Worse, this router wouldn’t accept more memory. At about six in the morning, I finally got an admission from the router vendor that they could not help me.

I hung up the phone. The first client who had threatened departure would be checking in at seven thirty AM. I had slept four hours of the last forty-eight, and had spent most of that time under fiendish levels of emotional stress. I had already emptied my stash of quarters for the soda machine, and had pillaged a co-worker’s desk for his. The caffeine and adrenaline that had gotten me to the office had long since worn off, and further doses of each merely slowed my collapse. We had support contracts on every piece of equipment, and they were all useless. All the hours of work my team and I had put in left me with absolutely nothing.

I made myself sit still for two minutes simply focusing on breathing, making my head stop sliding around loose on my shoulders, and ignoring the loud ticking clock. What could be done in ninety minutes — now only eighty-eight?

I really had one only option. If it didn’t work, I would either lay someone off or file for unemployment myself.

6:05 AM. I started downloading the OpenBSD install floppy image then grabbed a spare desktop machine, selecting it from amongst many similar machines by virtue of it being on top of the pile. The next few minutes I alternated between hitting the few required installation commands and dismantling every unused machine unlucky enough to be in reach to find two decent network cards.

By 6:33 AM I had two Intel EtherExpress cards in my hands and a virgin OpenBSD system. I logged in long enough to shut the system down so I could wrench the case off, slam the cards into place, and boot again. Even early versions of PF included all sorts of nifty filtering abilities, all of which I ignored in favor of the newly-integrated traffic-shaping functions. By 6:37 AM I was wheeling a cart with a monitor, keyboard, and my new traffic shaper over to the rack.

Then things got hard. I didn’t have a spare switch that could handle our Internet bandwidth. The router rack was jammed to overflowing, leaving me no place to put the new shaper. I lost almost half an hour finding a crossover cable, and when I discovered one it was only two feet long. The router, of course, was mounted in the top of the rack. About 7:10 AM, I discovered that if I put the desktop PC on end, balanced it on an empty shipping box, and put the box on the mail cart, the cable just reached the router. I stacked everything so it would reach and began re-wiring the network and reconfiguring subnets.

I vaguely recall my manager coming in about 7:15 AM, asking with taut calmness if he could help. If I remember correctly, as I typed madly at the router console I said “Yes. Go away.”

At 7:28 AM we had an OpenBSD traffic shaper between the hosting area and our router. All the client applications were reachable from the Internet. I collapsed in my chair and stared blankly at the wall.

While everything seemed to work, the proof would be in what happened as our offending site started its daily business. I watched with growing tension as that client’s network traffic climbed towards the red line that indicated trouble. The traffic grew to just short of the danger line — and flatlined. Other clients called, happy that their service was restored to its usual quality. (One complained that his site was still slow, but it turned out that bandwidth problems had masked an application problem.) The problem client complained that their web site now ran even slower than before, to which we offered to purchase more bandwidth if they’d agree to buy it.

I taped a note to the shipping box that said “Touch this and I will kill you,” staggered to my car, and by some miracle got home.

Shortly afterwards, I had two new routers and new DS3s. The racks were again clean. The decrepit desktop machine was replaced by two rack-mount OpenBSD boxes in a live-failover configuration, protecting our big-name commercial firewall as well as shaping traffic. And I now keep a crossover cables in a variety of lengths.

Should we have had traffic shaping in place before selling service? Absolutely. As with any startup, though, our hands were full fixing the agonies of the moment and less on the future.

If I had started with OpenBSD, I would have had a much better night.

(Want more OpenBSD? Check out my book Absolute OpenBSD.)

A modern hand scraper and wire brush can strip peeling, mildewy paint from a concrete basement wall almost easily — at least, much easier than when I was a kid and had to do the same job with a pointed stick and piece of chalk. The equipment comes with warnings in big black letters. “Wear Goggles!” “Wear Gloves!” “May Sever Fingers!” And so on. You don’t want to get a flying paint chip in your eye.

Unfortunately, it doesn’t come with a warning that says “Keep Mouth Shut.”

Describing the taste of a hundred-year-old mildewed paint chip as “Lovecraftian” would leave me without adequate vocabulary to describe the texture.

The moral is: when you need to shut up and do the job, don’t forget the “shut up” part.

On the NetBSD blog you’ll find an announcement that Microsoft has donated working code to support an experimental hardware platform to NetBSD.

Microsoft has a mixed relationship with open source software. There’s the perennial discussions about Windows using BSD’s TCP/IP stack, .NET for FreeBSD, Microsoft buying and killing a NetBSD-based phone, and any amount of blather ranging from the absurd to the paranoid. What makes this different?

First, it’s a gift. No strings attached — the BSD license doesn’t support strings. Copyright has been assigned to the NetBSD Foundation. It’s ours now, and there’s nothing Microsoft — or anyone — can do to take it back.

Second, the extensible MIPS hardware can be reconfigured in software to support application-specific tasks. This is cool. I’m sure that someone will tell me that this was done twenty years ago and that the prior work has been unfairly ignored since, and someone else will tell me that this is really no big deal, but it sure sounds interesting to my uneducated ears.

Third, NetBSD support will help get extensible MIPS running on other BSD platforms, and to a lesser extent on other operating systems. If the hardware ever becomes widespread, that is.

I doubt that this means any sea change in Microsoft’s relationship with open source. This code is of limited use today, given the scarcity of hardware. Microsoft Research offering eMIPS patches would not surprise me, but there’s a difference between cooperation in research and cooperation anywhere else.

As a special Christmas present to myself, I’m solving a problem and making a positive improvement to my environment.  Using Perl and gnuplot.  I’m not going to share the actual code, for two reasons:  one, it’s very specific to an in-house problem, and two, I use a programming technique I call “iterative petulance.”

So instead, here’s something for the holidays:  land mines.

Land mines are bad.  Land mines that we’ve left lying around are really bad.  Clearing abandoned land mines is hard, dangerous, and expensive work.  My new favorite charity clears these landmines inexpensively and safely… using rats.  No, you don’t herd rats across minefields.  Hero Rats are trained to smell explosives, and are too small to set off the mines.

Most of us have too much stuff.  Why not adopt a rat for someone instead of giving them yet more stuff?

Hopefully, I’ll be doing something technically interesting early next year.  In the meantime, happy holidays!

By now you’ve probably heard of the allegations Theo forwarded to the OpenBSD-tech mailing list about the FBI introducing back doors in early versions of the OpenBSD IPSec code.  I’d like to offer my opinion, in the spirit of the Christmas season:

“Bah, humbug!”

It’s possible, but unlikely.  Like me winning the lottery is unlikely.  I’d need to buy a ticket, and that isn’t going to happen any time soon.

The OpenBSD group examines every line of code that goes into their tree.  Any obvious back door would be caught.  Any  subtle back door would be fragile — so subtle that it probably wouldn’t survive the intervening ten years of code churn and IPSec improvements.  Maybe someone has an appliance based on, say, OpenBSD 2.8 or 3.2, which could have contained the back door.  If true, we need to know about it.  But those users need to upgrade anyway.

And the FBI?  Nope, don’t believe it.  Ten years ago, the FBI was having lots of trouble understanding the Internet.  The NSA, maybe.

Bugs?  Sure, there’s probably bugs.  I expect we’ll find some, now that many eyes have turned to the code.  Exploitable bugs?  Maybe.  But that’s not the same as a back door.

OpenBSD has claimed to be the best for many years.  That claim motivates people to take them down.  The claims have hopefully inspired many people to examine the current and historical IPSec stack.  Theo and company have done nothing to discourage such audits: they’ve even offered pointers on where to look.  If you’re a programmer looking to make a splash, you could do worse than to join in on auditing the code.  Finding the alleged back door would make your reputation.  And we can always use more IPSec hackers.

The real impact might be, as Jason Dixon points out, the cost in OpenBSD developer time.  You know that some of their committers are examining the IPSec code today, trying to find potential back doors.

I was amused to discover the connection between Wikileaks and BSD.

Apparently Julian Assange hung around the BSD community up until ten years ago, and has a few entries in the NetBSD fortune files.  (Search for Julian Assange in the file, or just click on the next link for the best ones.)    He lived in the house where Greg Lehey grew up, although many years after Greg had moved on.  Greg was interviewed for a story in the Australian news. They botched it.

If you think about it, you’d realize the connection must go deeper than that.  We all know about Osama bin Lehey.  Apparently the house where Greg was raised has that effect on people.  I do believe that Lovecraft wrote a story about that… and it will bug me until I can remember which story that was.

Apparently my NYCBSDCon presentation, BSD Needs Books, went over well.  I was just invited to reprise it at BSDCan on 13-14 May 2011.

So, what’s the critical difference between NYCBSDCon and BSDCan?  Both have great people.  Both have great presentations.  But there’s one critical point in NYC’s favor.

It’s 0.95km between the U of O Residences at BSDCan to the gelato shop. From the St. Marks Hotel in NYC to the gelato shop is less than 50 meters.  BSDCan has clearly fallen behind in the critical factor in North American BSD conferences.  I’m confident Dan (Mr. BSDCan) can figure out some way to shift the balance back to Ottawa, though.

Isilon is clever.  And they really want to give lots of their code back to the FreeBSD community.

New York Internet donated space, cooling, and power for an East Coast FreeBSD mirror.  Companies like Juniper and NetApp are donating hardware.  We will soon have an East Coast mirror of the West Coast datacenter, including package building facilities.  This will be cool.

Databases suck.  SQL is an abomination.  I knew this already, but it’s nice to have that opinion reinforced.  We could really use a data query language based on relational algebra.

George Rosamond put con finances on display during lunch.  NYCBSDCon made money this year.  The leftovers will be cut in four and split between OpenBSD, NetBSD, FreeBSD, and DragonFly.

pfSense rocks.  Once the next release is out, the team will turn its attention to IPv6.

And I’ve got to up and give my talk in a few minutes.  Those of you at the conference might as well go home now.

A few quick random things I picked up at day 1 of NYCBSDCon:

• Scheduled IPv4 depletion date:  119 days.  That’s when the last /8 is issued to a regional NICs.  Many of the remaining IPv4 /8 blocks are “poisoned,” and receive garbage traffic immediately upon announcement.
• Hudson River Trading is hiring FreeBSD folks.  They gave away 1GB USB key/bottle openers, so they clearly understand the sysadmin mentality.
• You want to take the BSD Associate Cert as soon as possible.
• Don’t confuse George with George.  George doesn’t like that.  Fortunately, George doesn’t care, so you’ll only have to worry about George.
• The “Quest for the Next Generation FreeBSD Installer” is about to claim more developers.  You’d think people would learn.  (Don’t get me wrong, I wish them luck and I hope they succeed, but nobody’s ever had dinner after betting the grocery budget on a new FreeBSD installer.)
• Jeremy Reed is digging through the original BSD tapes and contacting every person named in the original source code to assemble a comprehensive BSD history.  BSD claims a long history, but Jeremy’s actually trying to document it while the original folks are still with us.  It will eventually be available as a book.  This is probably the most exciting thing I heard today, but then, I’m an academic at heart.
• And if any BSD folks live near Jason Dixon, he <i>really</i> needs to be dragged out of management.  Forcibly if necessary.  Possibly with methods involving tranquilizer darts, nets, and some sort of radio tags.  If you do this, be sure to post the video footage for the rest of us.

You can get here for tomorrow.  I know you can.