Author: Michael Lucas

Writer. Sysadmin. Owns this site.
Posted on

Ubuntu server 10.04 LTS diskless filesystem

A diskless server needs a copy of the operating system files, served from an NFS server.  The Ubuntu docs have a general-purpose tutorial on diskless systems, which suggests copying the files from your NFS server.  My NFS servers are not Ubuntu boxes.  Also, I don’t want to copy from a live system; too many things can happen.  I want a set of Ubuntu server files that I can use to deploy a functional server in a known good state, that complies with the requirements of my environment.  And I need to script it, so I can boot and update my “golden image” server and easily reproduce the same file set. And I want all the routine changes taken care of automatically.

This problem isn’t hard, but I’ve spent a fair amount of time building and rebuilding diskless systems lately, so you get to hear about it.

Install an actual Ubuntu system.  I prefer to install on a virtual machine.  This will become your “golden image.”  When the Ubuntu installer asks for a machine profile, choose OpenSSH server.

  • apt-get update && apt-get upgrade
  • Install required software, such as emacs, tcsh, and configure .
  • install portmap and nfs-common.
  • Install and configure LDAP auth and sudo against LDAP
  • Install and configure ufw.  I’ve seen many attacks against Ubuntu boxes lately, and highly recommend very restrictive firewall rules.  Do not let the world talk to your Ubuntu servers!
  • Make a VM snapshot of your base image, so you can revert to this core functionality
  • Install anything else required to make this a nice clean template for the purpose of this server.

Now mount a directory on another server on the clean server’s /mnt via NFS and tar up the server.

# cd /
# tar -cvpf /mnt/ubuntu1004.tar --one-file-system .

Wait.

The resulting tarball has a few problems.  I don’t want the diskless hosts to all have the same SSH keys, so those files need to be removed. Ubuntu caches the MAC address of attached NICs to maintain consistent interface names across reboots. This cached MAC address will be wrong for the diskless machine. The existing interface configuration will not work on a diskless machine (see below).  Finally, the fstab is wrong for any diskless machine.  The machine will get its hostname from DHCP, rather than from a file.  I therefore remove the troublesome files from the tarball.

# tar --delete -f /mnt/ubuntu1004.tar ./etc/ssh/ssh_host_rsa_key ./etc/ssh/ssh_host_rsa_key.pub ./etc/ssh/ssh_host_dsa_key ./etc/ssh/ssh_host_dsa_key.pub ./etc/udev/rules.d/70-persistent-net.rules ./etc/fstab ./etc/network/interfaces ./etc/hostname


The difficult file is /etc/network/interfaces.  I don’t want to use the server’s network configuration.  My test server boots from either DHCP or with a static IP, and neither will work for a diskless server.  A diskless server needs an /etc/network/interfaces like this:

auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual

I want to replace the existing ./etc/network/interfaces with one of my own choosing.  Tar won’t let you replace a file in an existing archive, but it will let you add another file of the same name.  I change to a config directory and add this file to my tarball.  Similarly, I need a blank etc/fstab.  I create a fake etc directory in another location, touch etc/fstab, and create a suitable etc/network/interfaces.

# tar --append -f /mnt/ubuntu1004.tar etc/network/interfaces etc/fstab

To use this file, log into NFS server, go to the mount point for the diskless system, and run:

# tar -xpf /path/ubuntu1004.tar

The machine will then boot, is easily cloned, built to my standards, and the only customization needed is to run dpkg-reconfigure openssh-server.

As I installed on a virtual server I can snapshot the golden image and build custom filesystems for different purposes.

Lots of long commands?  Yep.  This basically screams “8-line shell script, please.”  It’s a pretty trivial script, but if you’ve made it this far, you’re either interested in what I’m doing or astonished at my inanity.  In either case, you should get the script too.

#!/bin/sh

mount nfs1:/tmpmount /mnt
cd /
tar -cvpf /mnt/ubuntu1004.tar –one-file-system .

tar –delete -vf /mnt/ubuntu1004.tar ./etc/ssh/ssh_host_rsa_key ./etc/ssh/ssh_host_rsa_key.pub ./etc/ssh/ssh_host_dsa_key ./etc/ssh/ssh_host_dsa_key.pub ./etc/udev/rules.d/70-persistent-net.rules ./etc/fstab ./etc/network/interfaces ./etc/hostname

cd /home/mwlucas/fakeroot
tar –append -f /mnt/ubuntu1004.tar etc/network/interfaces etc/fstab

Yes, this shell script is a good example of fault-oblivious computing. But it suits my minimal needs, and performs the same task the same way every time.

Posted on

“Page Cannot Be Displayed” and Internet Explorer

I detest this IE error message, especially when a user calls to complain that a Web site is down. Internet Explorer deliberately hides actual HTTP error messages on the grounds that the Web offers unfriendly but useful error messages.  Apparently this generic message is much less likely to cause the user to flee in terror from insanity-inducing text such as “404 – Page Not Found.”  They effectively shift the induced sanity from the end user to the sysadmin.

There’s a way to turn off this generic friendly message and replace it with the actual error.  It’s under Tools-> Internet Options -> Advanced -> Browsing -> Show friendly HTTP error messages.  Uncheck this and restart the browser to get user-hostile but troubleshooting-friendly error messages.

Every time I need this, I have to scramble to find it.  Perhaps now that I’ve documented this, I’ll remember where it is.  But I doubt it.

On an unrelated note:  tomorrow is the Thanksgiving holiday in the US.  I’d like to remind my readers that the holiday buffet is not a challenge, and that leaving food uneaten is not a threat to your masculinity (or femininity, or whatever).

Posted on

Firewalling diskless Ubuntu

I have diskless Ubuntu 10.04 servers sitting naked on the Internet.  They’re for internal use only, but I don’t have a firewall in that facility, so any firewalling must be done on the host itself.  Ubuntu includes UFW, the “uncomplicated firewall,” a front end to iptables.  I don’t know how anything can claim to make iptables uncomplicated, but I suppose nobody would use the tool if they called it “less appalling firewall.”

These servers need to be able to contact the Internet, to get updates and such, but nobody except myself and my coworkers need to access these servers. The coworkers and I only come from a limited range of IP addresses.

On a disk-based server, I would define rules in UFW and then run ufw default deny incoming, much like this:

# ufw enable
# ufw allow from 10.0.1.0/24
# ufw allow from 172.16.5.0/24
# ufw default deny

If you do this on a diskless Ubuntu server, the system loses disk — even if you have a rule that specifically permits access to the diskless server. The obvious thing to try is to rip out the “default deny” and replace it with a rule to block unwanted traffic at the end.

# ufw deny from 0.0.0.0/0

Your resulting rules look like this:

# ufw status
Status: active

To                         Action      From
--                         ------      ----
Anywhere                   ALLOW       10.0.1.0/24
Anywhere                   ALLOW       172.16.5.0/24
Anywhere                   DENY        Anywhere

This looks like it should work.  I attempt to connect to the SSH server from an IP not in the permitted list, however, and can connect.  It’s not blocking traffic from denied hosts.  Huh?

Go to the file that contains the user rules, /lib/ufw/user.rules.  This is actually a script to feed to iptables. There are several lines like this, one for each block of management addresses:

### tuple ### allow any any 0.0.0.0/0 any 10.0.1.0 in
-A ufw-user-input -s 10.0.1.0 -j ACCEPT

My last rule, however, looks different.

### tuple ### deny any any 0.0.0.0/0 any 0.0.0.0/0 in
-A ufw-user-input -j DROP

The “all other IP addresses” is probably implied in that last rule, but… it really couldn’t be that simple, could it?  I edit the script to explicitly specify the source IP addresses:

-A ufw-user-input-s 0.0.0.0/0 -j DROP

and reboot.

And yes, it is that simple.  The firewall comes up at boot.  ufw status displays exactly the same rules as before.  But now, I can only connect from my management IP addresses.

The problem with tools that make things “uncomplicated” is that rather than removing the underlying complexity, they hide it. I probably need to break down and learn iptables, but I think I’d rather figure out how to get these hosts behind a PF box.

Posted on

another brush with glory

One of my friends, SF writer Colin Harvey, just had his second mass market paperback hit the shelves.  Damage Time is a post-peak-oil police novel set in New York City.  The chilling bit is how he successfully combines the ideas of memory extraction with social networking.  And murder, of course.  (Where you have police, you get murder. Without police, people just get killed.)

If you look in the acknowledgments, you’ll find the line “and Michael Lucas hunted cliches relentlessly…”  That’s me.  I’ve got my name on another book!  Well, okay, in another book.  Close enough.  Sort of.  And apparently I’m relentless, too.  Maybe that’ll encourage me to get out of bed in the morning.

If you read SF, I highly recommend Damage Time.  Exciting, gritty, stolen memories, appalling and believable.

Posted on

I will be at BSDCan

Apparently my NYCBSDCon presentation, BSD Needs Books, went over well.  I was just invited to reprise it at BSDCan on 13-14 May 2011.

So, what’s the critical difference between NYCBSDCon and BSDCan?  Both have great people.  Both have great presentations.  But there’s one critical point in NYC’s favor.

It’s 0.95km between the U of O Residences at BSDCan to the gelato shop. From the St. Marks Hotel in NYC to the gelato shop is less than 50 meters.  BSDCan has clearly fallen behind in the critical factor in North American BSD conferences.  I’m confident Dan (Mr. BSDCan) can figure out some way to shift the balance back to Ottawa, though.

Posted on

mod_security on FreeBSD

The constant stream of referrer spam isn’t sufficiently annoying; no, now worms constantly nibble at my WordPress install.  I could avoid worrying about this by, say, having a third party host my content and control my work, but if I did that I’d get a punch on both my geek card and my writer card.  And I still wouldn’t know who is linking to me.  Some of the referral spam I get hits 10-15 times a day, flooding actual links.

Fortunately, Apache’s mod_security can help lock down my server.  While you’ll find tutorials on using mod_security to stop referrer spam, mod_security can do much more.  Here I’m installing mod_security on my FreeBSD server running Apache 2.2.

# cd /usr/ports/www/mod_security
# make all install clean

Look in /usr/local/etc/apache22/Includes afterwards.  You’ll find the file mod_security2.conf and the directory mod_security2.  Initially, mod_security is loaded into Apache but doesn’t block anything.  Go into the mod_security2 directory and edit the main config file, modsecurity_crs_10_config.conf.  Change the SecRuleEngine to On, and create a SecDataDir, like so:

SecRuleEngine On
SecDataDir /var/run/modsecurity

You’ll need to create the security data directory and make it writable by Apache.  Then restart Apache.

# mkdir /var/run/modsecurity
# chown www:www /var/run/modsecurity
# apachectl restart

Now test your Web server, and verify that it still functions.  Bad Web applications can trip over mod_security2.  If your Web app fails, I’d suggest talking to the vendor about why your application doesn’t work securely.

If your site still works with mod_security2, you can start to block referrers that bug you.  In the mod_security2 directory, create the file referer.conf for rules to block bogus referrers.  The rule has this general syntax:

SecRule REQUEST_HEADERS:REFERER “REGEX” deny,log,status:500

mod_security will evaluate each incoming request by its header.  If the referrer matches the regular expression in quotes, the browser will return a 500 error.  The sample rules below show a small slice of the things I’m blocking.

…
SecRule REQUEST_HEADERS:REFERER “write\-a\-resume” deny,log,status:500
SecRule REQUEST_HEADERS:REFERER “wigmall” deny,log,status:500
SecRule REQUEST_HEADERS:REFERER “windowsphone” deny,log,status:500
SecRule REQUEST_HEADERS:REFERER “windows\-phone” deny,log,status:500
SecRule REQUEST_HEADERS:REFERER “zune” deny,log,status:500

It’s possible that this would block legitimate traffic, but I have a hard time imagining being linked from a weight loss or Windows Phone site.  It’ll take a while to accumulate a list of suitable regexes for my site.  And it’s a limited technique — I’m enumerating badness. But mod_security also protects me against the various WordPress worms, and it can also block traffic from addresses on an RBL. I’ll do that at a later date.

UPDATE: Your SecRule should not include the “log” keyword. See the later posting here.

UPDATE2: more here.

Posted on

Things I Learned at NYCBSDCon, day 2

Isilon is clever.  And they really want to give lots of their code back to the FreeBSD community.

New York Internet donated space, cooling, and power for an East Coast FreeBSD mirror.  Companies like Juniper and NetApp are donating hardware.  We will soon have an East Coast mirror of the West Coast datacenter, including package building facilities.  This will be cool.

Databases suck.  SQL is an abomination.  I knew this already, but it’s nice to have that opinion reinforced.  We could really use a data query language based on relational algebra.

George Rosamond put con finances on display during lunch.  NYCBSDCon made money this year.  The leftovers will be cut in four and split between OpenBSD, NetBSD, FreeBSD, and DragonFly.

pfSense rocks.  Once the next release is out, the team will turn its attention to IPv6.

And I’ve got to up and give my talk in a few minutes.  Those of you at the conference might as well go home now.

Posted on

Things I Learned at NYCBSDCon, Day 1

A few quick random things I picked up at day 1 of NYCBSDCon:

  • Scheduled IPv4 depletion date:  119 days.  That’s when the last /8 is issued to a regional NICs.  Many of the remaining IPv4 /8 blocks are “poisoned,” and receive garbage traffic immediately upon announcement.
  • Hudson River Trading is hiring FreeBSD folks.  They gave away 1GB USB key/bottle openers, so they clearly understand the sysadmin mentality.
  • You want to take the BSD Associate Cert as soon as possible.
  • Don’t confuse George with George.  George doesn’t like that.  Fortunately, George doesn’t care, so you’ll only have to worry about George.
  • The “Quest for the Next Generation FreeBSD Installer” is about to claim more developers.  You’d think people would learn.  (Don’t get me wrong, I wish them luck and I hope they succeed, but nobody’s ever had dinner after betting the grocery budget on a new FreeBSD installer.)
  • Jeremy Reed is digging through the original BSD tapes and contacting every person named in the original source code to assemble a comprehensive BSD history.  BSD claims a long history, but Jeremy’s actually trying to document it while the original folks are still with us.  It will eventually be available as a book.  This is probably the most exciting thing I heard today, but then, I’m an academic at heart.
  • And if any BSD folks live near Jason Dixon, he <i>really</i> needs to be dragged out of management.  Forcibly if necessary.  Possibly with methods involving tranquilizer darts, nets, and some sort of radio tags.  If you do this, be sure to post the video footage for the rest of us.

You can get here for tomorrow.  I know you can.

Posted on

NYCBSDCon is at our throats

NYCBSDCon is this next Friday, Saturday, and Sunday, 12-14 November 2010.  If you’re anywhere near NYC, you need to attend.  If you’re not anywhere near NYC, you need to get to somewhere near NYC, and then attend.

I’ll be speaking on Sunday.  This talk could be subtitled “How I Reduce Suckage in My Books.”  Writing decent tech books is a skill you can learn.  I can honestly say that the slides are done, but in truth is that I still need to reduce slide suckage.  I’ll be reducing suckage up until the moment I present.

Posted on

fixing ESXi “failed with error N7Vmacore15SystemExceptionE”

An ESXi server failed this morning.  As there’s a couple critical services on this piece of hardware, the power in the new data center isn’t up to where we want it yet, and the radio said it was snowing near the office, I drove in expecting to find some unspeakable power situation.  The power was fine, but the ESXi server was sitting at a panic screen.  Power cycle the machine.  It comes up, but none of the VMs start.  The vSphere client won’t connect.  The server Web page is blank.

Fortunately, tech support mode works.  Hit alt-F1, type unsupported, and enter the root password when asked.  Whenever I tried to connect to the server with vSphere, my “tail -f /var/log/messages” said something like:

Nov  4 23:35:09 Hostd: [2010-11-04 23:35:09.117 25233B90 warning 'Proxysvc Req00011'] 
Error reading from client while waiting for header: 
N7Vmacore15SystemExceptionE(Connection reset by peer)

This is not good.  No, not good at all.  I wanted to spend the day converting a machine from OpenSolaris to FreeBSD and installing my router for my new bandwidth.  Instead Fate has decreed today Wedgie Day.

Mailing list archives and forum posts showed that many people have had this problem.  Lots of the forums end with “did anyone ever solve this?”  A few people reinstalled ESXi to solve the problem.  A couple folks claimed it was a DNS issue.

Our DNS setup hadn’t changed, but I followed the advice and made the following changes.

  • In /etc/hosts, remove the real address for the machine and replace it with 127.0.0.1
  • Remove all DNS servers from /etc/resolv.conf

I rebooted.  The machine came up, and the VMs started.  Everything seems fine, but we’ll have to see what happens later.

I have no idea why this worked.  Three cheers for “occult IT”!  Sigh.