Running Ancient Rsync

Another “write it down so I don’t forget what I did” post.

Some of the systems I’m responsible for are file storage machines, running rsync 3.0 or 3.1 as a daemon. Every hour, an ancient Solaris machine sends files to it using rsync 2.3.1. The billing team uses these files to create bills.

Thursday, I rebooted the machine. And the rsync stopped working with:

rsyncd[3582]: rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsyncd[3582]: rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.0]

The rsyncd server hadn’t changed. No security patches, no package updates, no nothing.

We cannot change the software on the Solaris machine. It’s attached to a multimillion-dollar telco switch, and editing the software on it would invalidate the warranty. The whole point of buying a multimillion-dollar telco switch is so you get the warranty. If something goes wrong, a team of vendor experts descends on your facility with enough spare parts to rebuild the switch from the ground up. (Telephony is nothing like IT. Really.) I cannot use SSH to transfer the files. I do not administer this machine–actually, I don’t want to administer this machine. I’m so unfamiliar with warranties on operating systems that I would probably void it by copying my SSH public key to it or something.

The Solaris box is running rsync 2.3.1, which runs rsync protocol version 20. My systems use newer rsync, running protocol version 30 or 31.

Rsyncd isn’t easily debuggable. Packet analysis showed messages about protocol errors. The rsync FAQ has a whole bunch of troubleshooting suggestions. None of them worked. I ran rsync under truss and strace and painstakingly read system calls. I eventually sacrificed a small helpless creature in accordance with ancient forbidden rites under last weekend’s full moon.

After a few days of running through a backup system (an old but not quite ancient OpenSolaris box), I absolutely had to get this working. So: protocol errors? Let’s try an older rsync.

Rsync 2.9? Same problem. I saw myself progressively working my way through building older versions, solving weird problems one by one, and eventually finding something old enough to work. This is not how I wanted to spend my week. Given how well running FreeBSD 4 in a FreeBSD 10 jail works, I tried something similar.

The host ftp-archive.freebsd.org host releases of every FreeBSD version, including packages. FreeBSD 10 includes compatibility with FreeBSD back to version 4. I installed the compatibility libraries from /usr/ports/misc/compat4.

The oldest FreeBSD 4 rsync package I could find was 2.4.6, from FreeBSD 4.1.1. Original FreeBSD packages were just zipped tar files. I extracted the files and checked that the binary could find all its libraries.

# ldd rsync
rsync:
libc.so.4 => /usr/local/lib32/compat/libc.so.4 (0x2808a000)

If this was more complicated software, with more libraries, I’d have to track down the missing ones. Rsync is very straightforward, however.

I shut down the old rsync daemon and fired up the old one.

It worked.

I still want to know how a reboot broke this. I’m assuming that something changed and that I lack the sysadmin chops to identify it. It’s not the rsync binary, or libc; both have date stamps several months old.

I don’t recommend this, as older rsync has all kinds of security problems. These particular hosts are behind several layers of firewalls. If an intruder gets this far, I’m basically doomed anyway.

So: if you’re very very stuck, and the clock has run out, using really old software is an option. But it still makes my skin crawl.

Trying poo-DRE-eh — uh, poudriere

This is my poudriere tutorial. There are many like it. But this one is mine. I built mine with resources like the BSDNow tutorial and the FreeBSD Forums tutorial. While all poudriere tutorials are inadequate, mine is inadequate in new and exciting ways. I’m writing it for my benefit, but what the heck, might as well post it here. (If you read this and say “I learned nothing new,” well, I warned you.)

Your package building system must run the newest version of FreeBSD you want to support. I have 8, 9, and 10 in production, so my package builder needs to be FreeBSD 10 or newer. I’m using FreeBSD 11, because I’m running my package builder on my desktop. I upgraded this machine to the latest -current and updated all my packages and ports.

Poudriere works on either UFS or ZFS partitions. I have copious disk, so I’ll dedicating two of them to poudriere. (This means I’ll have to blow away my install later when I start experimenting with disks, hence this blog posting.) I use disks ada2 and ada3.

First, eradicate anything already on those disks.

# gpart destroy -F ada3
ada3 destroyed
# gpart destroy -F ada2
ada2 destroyed
#

Now I can create new GPT partitions. Each disk needs one partition for ZFS, covering the whole disk. You’ll find lots of discussion about partitioning disks with 512-byte sectors versus those with 4KB sectors, and the need for carefully aligning your disk partitions with the underlying sector size. The easy way around this is to duck the whole issue and assume 4KB sectors, use a null GEOM layer to align everything to the disk’s expectations, and ZFS the null layer. (If you KNOW the sector size of your disk, you can simplify the below, but remember, disks lie about their sector size just like they lie about geometry. It’s safest to gnop all the things.)

# gpart create -s gpt ada2
# gpart create -s gpt ada3
# gpart add -a 4k -t freebsd-zfs -l disk2 ada2
# gpart add -a 4k -t freebsd-zfs -l disk3 ada3
# gnop create -S 4096 /dev/gpt/disk2
# gnop create -S 4096 /dev/gpt/disk3
# zpool create poudriere mirror /dev/gpt/disk2.nop /dev/gpt/disk3.nop

I now have a filesystem to build packages on. Let’s get me a key to sign them with.

# mkdir -p /usr/local/etc/ssl/keys
# mkdir -p /usr/local/etc/ssl/certs
# cd /usr/local/etc/ssl
# openssl genrsa -out keys/mwlucas.key 4096
# openssl rsa -in keys/mwlucas.key -pubout > certs/mwlucas.cert

Installed the latest poudriere from /usr/ports/ports-mgmt/poudriere-devel. While the configuration file is lengthy, you don’t need to set many options.

ZPOOL=poudriere
FREEBSD_HOST=ftp://ftp.freebsd.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/poudriere
POUDRIERE_DATA=${BASEFS}/data
USE_PORTLINT=no
USE_TMPFS=yes
DISTFILES_CACHE=/usr/ports/distfiles
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/mwlucas.key
URL_BASE=http://storm.michaelwlucas.com/

I set the CHECK_CHANGED options because when I update my ports tree, I want to know about changed options before I build and deploy my packages. I set the URL_BASE so I can view the build logs on my web server.

The build process uses its own make.conf file, /usr/local/etc/poudriere.d/make.conf, where I set some very basic things. The only mandatory setting is WITH_PKGNG. You could also have a separate make.conf for each individual jail, but I want all of my packages consistent, so I only use the central file.

WITH_PKGNG="yes"
WITHOUT_X11="yes"
WITHOUT_HAL="yes"
WITHOUT_NLS="yes"

Get a ports tree just for poudriere. I could use the ports tree on the package builder, but it’s possible that ports on the builder might differ from what I want for the packages. It’s best to keep everything tidy.

# poudriere ports -c
====>> Creating default fs... done
====>> Extracting portstree "default"...
Looking up portsnap.FreeBSD.org mirrors... 7 mirrors found.
Fetching public key from your-org.portsnap.freebsd.org... done.
Fetching snapshot tag from your-org.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
...

Now create a jail to build packages in.

At first blush, you might give the jail any random name. Using the same jail standard as the FreeBSD package builder eases deployment, however. The official naming standard combines operating system, CPU architecture, operating system version, and word size, like so. For example, 32-bit FreeBSD 10 on Intel-type processors is freebsd:10:x86:32, while the 64-bit version is freebsd:10:x86:64. (See pkg-repository for details.)

FreeBSD’s official package builds occur on the oldest supported version of a release. Packages for 9-stable are built on 9.1, so I do:

# poudriere jail -c -j freebsd:9:x86:32 -v 9.1-RELEASE -a i386

Then create jails for the other two releases I support.

# poudriere jail -c -j freebsd:9:x86:32 -v 9.1-RELEASE -a amd64
# poudriere jail -c -j freebsd:10:x86:64 -v 10.0-RELEASE -a amd64

I only run the amd64 version of FreeBSD 10, because I don’t want to build i386 packages forever.

This takes a little while, so start it and walk away. I copied these lines into a shell script and went to lunch.

Note that these are not actual jails. They will not show up in jls(8). Technically, they’re chroots. (I’m not sure why poudriere calls them jails – maybe they were originally such but the need for a full jail went away, or they’re intended to become full jails later on. I’m sure there’s a perfectly sensible reason, however.) You can list your package-building jails only via poudriere.

# poudriere jail -l
JAILNAME VERSION ARCH METHOD PATH
freebsd:9:x86:64 9.1-RELEASE amd64 ftp /poudriere/jails/freebsd:9:x86:64
freebsd:9:x86:32 9.1-RELEASE i386 ftp /poudriere/jails/freebsd:9:x86:32
freebsd:10:x86:64 10.0-RELEASE amd64 ftp /poudriere/jails/freebsd:10:x86:64

Before building any packages, I want to update all the jails to the latest version. As I’ll need to do this before every package build, I script it. I also add the command to update poudriere’s ports tree.

#!/bin/sh

#update ports tree
poudriere ports -p default -u

#Update known builder jails to latest version
poudriere jail -u -j freebsd:9:x86:32
poudriere jail -u -j freebsd:9:x86:64
poudriere jail -u -j freebsd:10:x86:64

I must run this script every time I update my packages.

Now to determine which packages I want to build. In my case, I want to build only packages that I can’t get from official FreeBSD sources. This means things like freeradius and Apache with LDAP support and PHP with Apache support. Simple enough. I created /usr/local/etc/poudriere.d/pkglist.txt containing:

#web services - need LDAP & Apache PHP module
www/apache22
lang/php5
#network - need LDAP & SMB
net/freeradius2
net/openldap24-server

Now the fun part: setting the build options for these ports.

# poudriere options -cf pkglist.txt

This takes you into a recursive make config for all of your selected packages. If you know exactly which ports you must configure to get a properly built package, you could specify ports by name. Unfortunately, I always forget to add LDAP to some dependency, so I walk through all the configurations adding my various options.

You can now build your packages. I want to update all of my packages simultaneously, so I wrote a trivial shell script to build packages.

#!/bin/sh

#Update known builder jails to latest version

poudriere bulk -f /usr/local/etc/poudriere.d/pkglist.txt -j freebsd:9:x86:32
poudriere bulk -f /usr/local/etc/poudriere.d/pkglist.txt -j freebsd:9:x86:64
poudriere bulk -f /usr/local/etc/poudriere.d/pkglist.txt -j freebsd:10:x86:64

Walk away. Or, if you prefer, set up a web server so you can see the build progress and deliver packages to clients. I used apache22 and created /usr/local/etc/apache22/Includes/poudriere.conf containing:

NameVirtualHost *:80


ServerAdmin mwlucas@minetworkservices.com
DocumentRoot /poudriere/data
ServerName storm.blackhelicopters.org


Options Indexes FollowSymLinks
AllowOverride AuthConfig
Order allow,deny
Allow from all

Yes, I could have hacked up httpd.conf to set DocumentRoot, but I prefer to leave package-created files alone if possible.

Let’s turn to the client while this builds. The client needs a repository configuration file and a copy of the build certificate. Here’s the configuration for my local repository, named mwlucas.

mwlucas: {
url: "http://storm.blackhelicopters.org/packages/${ABI}-default/",
mirror_type: "http",
signature_type: "pubkey",
pubkey: "/usr/local/etc/ssl/certs/mwlucas.cert",
fingerprints: "/usr/share/keys/pkg",
enabled: yes
}

By using the standard jail names, I was able to use the ${ABI} variable in my repository path. This saves me from needing to update my repository configuration every time I upgrade.

I copy my certificate to the directory specified in the pubkey option.

You should be able to browse to the URL given as your repository. If the URL doesn’t work, you misconfigured your web server.

Check your repository configuration with pkg -vv. You should see all configured repositories.

# pkg -vv
...
Repositories:
FreeBSD: {
url : "pkg+http://pkg.FreeBSD.org/freebsd:10:x86:64/latest",
enabled : yes,
mirror_type : "SRV",
signature_type : "FINGERPRINTS",
fingerprints : "/usr/share/keys/pkg"
}
mwlucas: {
url : "http://storm.blackhelicopters.org/packages/freebsd:10:x86:64-default/Latest/",
enabled : yes,
mirror_type : "HTTP",
signature_type : "PUBKEY",
fingerprints : "/usr/share/keys/pkg",
pubkey : "/usr/local/etc/ssl/certs/mwlucas.cert"
}

Two repositories? Good. But can you actually access the repository? A pkg update should pull down the digests for your new repo.

# pkg update
Updating repository catalogue
digests.txz 100% 1928 1.9KB/s 1.9KB/s 00:00
packagesite.txz 100% 10KB 10.0KB/s 10.0KB/s 00:00
Incremental update completed, 23 packages processed:
0 packages updated, 0 removed and 23 added.

A quick check will verify that the private repo has 23 packages.

Now for the weakest part of pkgng: actually using your repository for select packages. If I built everything, I’d just disable the FreeBSD repository. But I want to only build packages that differ from the default.

pkg searches for packages in each repository in order. At the moment, repositories appear in alphabetical order. I could rename my repository so that it appears before FreeBSD. pkg would search my repo for packages, and if they didn’t exist there check the FreeBSD repo.

This would be ideal. But alphabetical repository ordering is not guaranteed. The only way to ensure the packages install from the correct repository is to tell each individual FreeBSD server where it should get specific packages. This kind of sucks, but it’s still an improvement on pkg_add. (I hear that repository handling should improve in future versions of pkg.)

Use the -r flag to specify a repository with pkg. For example, let’s search my private repo for an Apache package.

# pkg search -r mwlucas apache
apache22-2.2.26

The package is there. Excellent. Install it.

# pkg install -r mwlucas apache22
Updating repository catalogue
The following 8 packages will be installed:

Installing expat: 2.1.0 [mwlucas]
Installing perl5: 5.16.3_7 [mwlucas]
Installing pcre: 8.34 [mwlucas]
Installing openldap-client: 2.4.38 [mwlucas]
Installing gdbm: 1.11 [mwlucas]
Installing db42: 4.2.52_5 [mwlucas]
Installing apr: 1.4.8.1.5.3 [mwlucas]
Installing apache22: 2.2.26 [mwlucas]

The installation will require 94 MB more space

19 MB to be downloaded

Proceed with installing packages [y/N]: y

Take careful note of this list of dependency packages installed from your repository.

Here you have to decide how you want to upgrade your server. All packages that you want to upgrade from your repo need to be marked as such. How many of these dependency packages must be upgraded via your repo? That’s a good question. If you took note of which packages you had to change the configuration of, way back in the beginning, you could label only those changed packages as requiring your repo. I didn’t do that. If this was a brand-new machine I would install packages from my repo first and mark everything installed as requiring my repo. I didn’t do that. Instead, I’m going to mark all packages required by apache22 as belonging to my repo, much like this.

# pkg annotate -Ay apache22 repository mwlucas
apache22-2.2.26: added annotation tagged: repository
# pkg annotate -Ay pcre repository mwlucas

Repeat this for each package installed by this package install.

The repository tag appears when you run pkg info on a specific package.

# pkg info pcre | grep repository
repository : mwlucas

pkg upgrade will now only use your repository for the specified package.

Is this long and complicated? Sure. But it beats the snot out of maintaining three separate package repositories under the old package system.

Installing FreeBSD 10 to ZFS with a script

Well, partially scripted, that is.

For installing large numbers of identical machines, proceed directly to the PC-BSD installer. It’s easy to configure, very reliable, and generally just rocks. If you’re accustomed to automatic installers like Kickstart, you’ll find the PC-BSD installer trivially easy.

I frequently have to install non-identical machines for special purposes, such as testing or unique file stores or EDI. Most of these are virtual machines. It seems that ZFS filesystems compress really really well, simplifying backing up the VMs.

And there, the FreeBSD 10 installer’s ZFS features don’t quite cut it. The installer lets you create a single large ZFS without leaving the GUI. I want a more complicated ZFS setup, based on the FreeBSD Root on ZFS wiki page. This process involves a whole lot of typing. I normally install servers when I’m too brain dead to do any real work, so I need to minimize the opportunity for errors.

Fortunately, you can script all the disk and ZFS setup. Here’s my script. If it looks familiar, well, it should: it’s ripped raw and bleeding from the wiki instructions and wrapped up with /bin/sh -x. (I use -x because I want to see how the script runs.)

#!/bin/sh -x

#Auto-divides a ZFS install.
#ZFS permissions stolen from
#https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/9.0-RELEASE

#edit:
#disk device name
#parameters for your zpool type
#your pool name
#swap space

#we're installing
sysctl kern.geom.debugflags=0x10

gpart destroy -F vtbd0
gpart create -s gpt vtbd0
gpart add -s 222 -a 4k -t freebsd-boot -l boot0 vtbd0

gpart add -s 1g -a 4k -t freebsd-swap -l swap0 vtbd0
gpart add -a 4k -t freebsd-zfs -l disk0 vtbd0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 vtbd0
gnop create -S 4096 /dev/gpt/disk0

kldload zfs
zpool create -f -o altroot=/mnt -O canmount=off -m none zroot /dev/gpt/disk0.nop

zfs set checksum=fletcher4 zroot
zfs set atime=off zroot

zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default
zfs create -o mountpoint=/tmp -o compression=lzjb -o setuid=off zroot/tmp
chmod 1777 /mnt/tmp

zfs create -o mountpoint=/usr zroot/usr
zfs create zroot/usr/local

zfs create -o mountpoint=/home -o setuid=off zroot/home
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages

zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src
zfs create zroot/usr/obj

zfs create -o mountpoint=/var zroot/var
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
chmod 1777 /mnt/var/tmp
zpool set bootfs=zroot/ROOT/default zroot

cat << EOF > /tmp/bsdinstall_etc/fstab
/dev/gpt/swap0 none swap sw 0 0
EOF

exit

How do I use this?

When the installer asks me how I want to partition the disk, I exit to the shell, configure the network, and get the script onto the target system.

# dhclient vtnet0
# cd /tmp
# fetch http://www-old.michaelwlucas.com/zfsinstall.sh
# chmod 755 zfsinstall.sh
# ./zfsinstall.sh

Sit back and watch it run. When the script finishes, exit from the shell and let the installer unpack the files. When you’re offered a shell to perform post-install configuration, take it and run the post-install ZFS setup commands.

# mount -t devfs devfs /dev
# echo 'zfs_enable="YES"' >> /etc/rc.conf
# echo 'zfs_load="YES"' >> /boot/loader.conf
# zfs set readonly=on zroot/var/empty

Reboot into your fine-grained ZFS filesystem installation.

To use this script yourself, you’ll need to check the disk device name and the type of zpool you want to create. But this will hopefully get you started.

I would really like to see the default FreeBSD installer create finer grained ZFS filesystems. I’m told that day is coming.

2013 Failures and 2014 Goals

I set goals for 2013. And I failed to meet them. I promised three short nonfiction books, Absolute OpenBSD 2nd edition, and a novel. You got AO2e and two short nonfiction books, DNSSEC Mastery and Sudo Mastery.

While setting goals is important, exploring why you fail to meet those goals is just as important. Driving factors behind these goals boil down to three things.

  • These were pretty ambitious goals
  • Traveled to EuroBSDCon in September
  • January’s emergency appendectomy
  • I knew this was ambitious beforehand, but decided to try for it anyway. So, the first I accept as my own inability to realistically predict what I can do.

    I spent two weeks in Europe, both for EuroBSDCon and meeting with other writers and publishers. If I had to fly for eight hours one way (which I detest), and shift my body clock (which I find very difficult), I was going to make the trip worthwhile. But between preparing for teaching at EuroBSDCon, physical preparations for the trip, and recovering from the trip (both physically and real life), that cost me at least a month.

    You cannot predict something like an appendolith. That’s life. I didn’t merely have an appendolith, though. I had fever and infection and all sorts of horrible ghastly things. Proper recovery took months. Plus, general anaesthesia is insidious. Even when you wake up, it muddles your brain for weeks or months afterwards.

    When life derails your goals, you get back up as soon as you can and get back on track. Maybe you can’t complete the entire goal, but you can sure do a whole bunch of it. Or maybe the deadline slips into the next year. Whatever you do, you don’t quit.

    So: I failed.

    With those things in mind, let me set some goals for 2014. I already let part of this out at NYCBSDCon, so the rest of you might as well know.

    1) I will write at least three short nonfiction books. At least one will be on OpenBSD, at least one will be on FreeBSD. At least two will see print by the end of the year.

    2) Last year’s novel will get out of my house. A couple of my author friends are encouraging me to run the novel through a publisher and have offered introductions. Their faith in my work is sincerely touching. I’m inclined to self-publish, but am keeping an open mind. We’ll see what happens. (I waited to publish this list until I finished the first draft, for those who wonder.)

    3) I’ll write at least 120,000 words of fiction. (See FAQ 9.)

    4) I will not change time zones for a conference. EuroBSDCon was great, and I’m sure that the Sofia conference will be just as grand, but that kind of travel messes me up too badly to write. I’ll be at BSDCan, but this year I’m taking the train. Because I really, really abhor flying.

    5) I’m a candidate for my dojo’s red sash test this year. If selected, I will do my best to pass. This means much practice and sweat, as the test lasts several hours. For example, my green sash test included over four hundred falls. The falling isn’t bad, but getting up again gets pretty rough. The red sash test is worse.

    My deadline for these goals in February 2015. Because my birthday is in February. Using my personal year for goals always feels better than using the calendar year.

    In a more general sense:

    I’m starting a series of short FreeBSD books, each dedicated to a single topic. Which topics will I cover? Whatever I’m working with at the moment, that’s holding still long enough for me to write about it. For example, at this moment it doesn’t make sense for me to write a book about pkgng, because pkgng is developing quickly.

    Eventually, I’ll create enough FreeBSD content to “remix” into a big FreeBSD book, probably a 3rd edition of Absolute FreeBSD.

    The small books will use the 6×9 form factor, and all be about the size of SSH Mastery. People have taken well to this size of book at the $10 ebook/$20 print price point.

    This will also let me judge which material should go into a big book. If nobody buys, say, a small FreeBSD virtualization book, it’s clear I shouldn’t put that topic into a big book, because nobody cares.

    Ideally, I’ll be able to produce a slipcase for a complete collection of small FreeBSD books. At this time, I’m planning to give them themed covers based on old pulp magazines, minus the blatant sexism and racism. (It’s been suggested by more than one person that I keep both elements but make them funny. It CAN be done, just as it is possible to make thoughtful, incisive, and honestly funny jokes about any other painful or horrifying topic. But it’s extraordinarily hard, especially for someone who looks utterly “privileged white male.” I choose to spend my energy elsewhere.) But Beastie as a hard-boiled private eye, Beastie swinging on a vine through the jungle, Beastie as the flying ace, and so on? I think that’s going to look fantastic.

    What will the OpenBSD book be? I have three ideas. I’ve caught wind of other OpenBSD books in progress, however. I need to meet with my fellow BSD authors at BSDCan 2014 and hash things out with them. It’s very important that we not step on each other’s else’s projects, especially when it’s simple enough to avoid with five minutes at the bar. That’s why I won’t do, say, a pfSense book — Chris and Jim have that territory covered quite well. I’m confident that at least one of my three ideas will be free, if for no other reason than we don’t have that many OpenBSD authors.

    I expect to let the FreeBSD Foundation have books at cost for PBS-style donation prizes. “Donate $100, and we’ll send you this $20 book!”

    I have a clever idea for using the OpenBSD book to support OpenBSD. Theo and I discussed it briefly at EuroBSDCon. I don’t know if it will actually work, mind you. But worst case, they’ll have my book in the OpenBSD bookstore, with proceeds going to OpenBSD. (For anyone who is wondering, Austin Hook is very very easy to work with. The hardest part of getting books to the OpenBSD bookstore is figuring out how to cram all the shipping information onto the CreateSpace web form, which is certainly not Austin’s fault.)

    So, is this a cynical scheme to get you to give me more money? No… and yes.

    You’ll have the option to give me any amount of money you wish, from zero up to over a hundred bucks. There’s a couple people that I suspect will buy every book, in every version. I suspect others will get a few of the small books. Others will wait for a big book. Some will buy all the small books just so they can fill a slipcase. This is about options. It’s about getting content into reader’s hands as quickly as possible.

    But if you want to give me money, I’m certainly not going to argue.

    The good news is, I now know exactly what an appendolith feels like. The next time my appendix blows up, I’ll jump on it at the earliest possible moment. Why, just today I’ve felt three twinges that might have been a faulty appendix. Catching these things early is the key to quick recovery, after all.

    Crazed Ferrets in a Berkeley Shower – 2014 edition

    With Richard Stallman’s recent raising of the flag against LLVM and Clang, I’ve heard a lot of people talking about how the “FreeBSD people are whining about the GPLv3 terrorizing them.”

    Back in 2000, I wrote an essay for Linux.com about why I like the BSD license. It’s actually stood up fairly well to the test of time, but it’s fourteen years old now. Times have changed. So have licenses.

    Normally I object to taking something with warts, dosing it with Compound-W, and sending it back out. The structure of the original still holds up pretty well, but experience and history has added new information and context. Any new essay I wrote would look an awful lot like this one.

    So, rather than try to write out my opinions from scratch, I’m taking advantage of the Internet and updating my previous essay.

    Crazed Ferrets in a Berkeley Shower (2014)

    When USL sued Berkeley over BSD UNIX, it would have been easiest and cheapest for the university to drop the BSD tapes down a deep well and forget the Computer Science Research Group ever existed. Instead, combining the academic tradition with Berkeley’s well-known liberal spirit, they fought to give their work to the world. In a decision that has caused more arguments than most in the open-source software world, they released the code under the BSD license.

    For anyone who has avoided free software licensing for the last five years, the BSD license is very simple: give the authors credit in the source code, and don’t sue the authors if it breaks. There is no requirement to distribute any modifications to anyone. Alternately, the GPL version 2 requires that source code and any modifications be made available when modified code is redistributed. GPL version 3 has additional terms addressing patent indemnification, Tivoization, and more.

    Every week sees a new argument on some public forum about how the GPL is free, and the BSD license isn’t. Someone responds that the BSD license is more free than the GPL. Eventually, someone drags out the word “communist,” someone else fires up “corporate exploitation,” and all hope for rational discussion vanishes like free Jolt cola on the trade show floor. It’s better than mud wrestling or Jerry Springer.

    Anyone familiar with my work knows that I’m on the BSD side of the fence. The GPL is not a bad thing — as an author I would argue that it’s poorly written as either a software license or a political manifesto, but the whole “share and share alike” ideal is grand. We try to teach our children the importance of sharing. I live sharing. I try to do it myself.

    The people who use the BSD license want something very different than the people who use the GPL. The license has different goals and different motivations than those of the GPL. The end goals require different licenses.

    Let’s go back in time, to the early days of BSD.

    Why would a university, especially such a famously liberal university, struggle to turn a top-notch operating system over to corporations for their pillaging pleasure? BSD UNIX gave thousands of developers the chance to innovate on a solid infrastructure. It raised the bar for minimal acceptable performance in a computer system.

    An organization can take a chunk of BSD-licensed code, shrink-wrap it, and resell it. They might make money. But if that’s all that they do, any competitor with an enhancement can take over their market and destroy them. Without real improvements and innovation, BSD-licensed code isn’t as useful as you might think.

    On the other hand, an organization can use BSD-licensed code in specific components, free programmers, and improve their products.

    If you want to build, say, a networked fax machine you can a) build an operating system from scratch, b) invest in an embedded OS such as QNX, c) use GPL-licensed code, or d) use BSD-licensed code. Most programmers would find building an operating system interesting the first time, tedious the second, and mind-numbingly dull the third. And the organization has to pay for the programmers’ time while he does it. Purchasing an operating system also raises the product’s cost. While the GPL is slowly becoming more accepted, the thought of giving away code still gives most corporate lawyers a bad case of rotating heads and the pea-soup-spews.

    By using BSD-licensed code, however, the programmer can spend his time working on the fax machine interface. He isn’t reinventing the wheel yet again. Competitors must improve, or die. The cost to the organization is reduced. The end-users get a device that is cheaper and more reliable.

    The organization must do lots of work to make this happen. Developing hardware and software for a fax machine, or any other networked device, is no easy task. All that the BSD-licensed code does is make a few select parts easy.

    Today, Intel has network appliances based on BSD. Apple has OS X. So do IBM, Netflix, and a lot of other household names. Entire businesses and careers have been built on BSD-licensed code. They’re big names now, but many of them started with some guy, a basement, and BSD. If the University of California, Berkeley, had not opened their code base under such a non-restrictive license, none of this would be possible. Such intelligent appliances would be far more expensive for both manufacturers and consumers.

    Organizations with special code, or some other kind of secret sauce, do not want to release that code to the world. If they’ve spent millions researching how to shuffle bits in a certain way, they want to get their money back. They will not use a GPL code base that requires they release their code to the world. It’s just not going to happen.

    While organizations are not required to return code to BSD-licensed projects, many do. Why?

    The biggest reasons are cheap debugging, wide test base, and ease of merging future releases.

    Open-source developers are quick to point out flaws in code, especially in a project that focuses so much attention on correctness. They will happily point out your boneheaded mistakes.

    Open-source end-users frequently run some of the hardest-working machines on the Net, and put donated code through stress tests the original authors never intended.

    Organizations don’t have to return improments to their source BSD project, but it is in their best interest — for purely commercial reasons. Hardware changes over time, as do operating systems, and this bites organizations in the future. Here’s a true story about a company’s involvement with FreeBSD. I’ve anonymized it and changed some minor details so as to not embarrass them further. (If you were there and watched the fun, please don’t name them.)

    Company X produced a product based on FreeBSD 2. It did well. They built a thriving business. Their product required changes to the FreeBSD kernel. Some of these changes were key intellectual property, while others were improvements to related systems that anyone could have done.

    They kept all their changes as a single massive patch set.

    Time passed. Hardware changed. Company X found that FreeBSD 2 wouldn’t work well for their new hardware. They needed to move to FreeBSD 4.

    All of the improvements they made needed to be ported to the new version. Their “secret sauce” had to be rewritten for the new kernel — but so did the less important changes. In many cases FreeBSD developers had improved the systems that Company X’s code patched, but in a way that was totally incompatible with their designs.

    Company X realized that if they interacted with the FreeBSD community more and sent their non-proprietary patches back to the project, future merges would be simpler. They could implement the next generation of their proprietary stuff in a way that would fit the FreeBSD development roadmap.

    They knew this… but didn’t manage to execute.

    Time passed. They had to redo their entire forward porting process for FreeBSD 6.

    I don’t know how many times these poor bastards went through this, but I’m glad to say they’re now consistent and regular FreeBSD contributors. (Pain is a wonderful teacher, but nobody wants to go to his school.)

    For me, knowing that Company X “gets it” and donates code back to FreeBSD makes me look upon them much more kindly. I know other organizations have taken BSD code and not returned anything. They’re not required to, but they don’t stick in my mind as one of the good guys. I also chuckle when I imagine them merging their code forward.

    Some people fear that the BSD license allows companies to take their work and make it proprietary. Source code isn’t a limited resource; any number of people can have it, use it, or improve it. The two programs that hold the Internet together, BIND and Sendmail, had BSD licenses for many years. If a company could seize control of one or another of these programs, the Internet would be their oyster and we would be their slaves. People have tried. People have spent lots of money trying to execute this nefarious master plan. It hasn’t happened. It hasn’t happened because it can’t.

    Even if a Big Evil Software Company (tm., pat. pend.) wanted to assimilate BIND, they would have to convince the current BIND users to pay for something they could still get for free. Any vital, but proprietary, improvement to BIND will be duplicated quickly by open-source developers. A closed-source development process cannot withstand the onslaught of open source when it comes to vital core functions, especially when you start with the same code base. Those of us watching the race would have to content ourselves with pointing, laughing, and buying the volunteers beer.

    Look at TCP/IP. It was originally implemented in BSD. The BSD implementation became the standard. Everybody uses it. If you write your own stack, it’s expected to be compatible with BSD TCP/IP. Or everyone calls you a loser and says your IP stack sucks. Which it does.

    Closed source companies can and do compete on the bells-and-whistles front. And people are willing to pay for the pretty point-and-click interfaces. Heck, more than once I’ve snarled obscenities while trying to make Sendmail relay mail for a specific type of client, and if you asked me at the right moment I would have plopped down a credit card to make the problem go away.

    Now, users without source code will have a hard time fixing bugs or adding features themselves. Most organizations today have trouble finding people who can make reports off their Access databases, let alone people who could fix device drivers. Organizations who have that talent use it. Most organizations fix problems by asking the vendor.

    And no code, whether GPLd or BSD’d, can make up for an organization that does not respond to its customers. Starting with known working code can give them a head start, but a non-responsive organization still won’t go far. You probably gripe about Microsoft and Apple, but the average user wants an operating system where they can easily install a kitten screen saver with extra spyware. The Windows interface might not work well, but it is easy. The average corporate customers want simple more than they want performance.

    From the developer’s viewpoint, code under the BSD license provides a certain “minimal acceptable level” of software quality. Can you write an IP stack better than the one in OpenBSD? If not, why bother? Isn’t there something you’d rather do than handle exceptions for SYNs, ACKs, and RSTs, then return every six months to deal with the latest denial-of-service attack?

    Hundreds of organizations have made this same decision; the BSD IP stack has found its way into countless products. Almost any network device that isn’t labeled Linux or System V uses BSD. Some products even combine them; the RTEMS real-time operating system, which is distributed under the GPL, incorporated the BSD IP stack.

    Things could have been very different.

    Imagine someone travels back in time to the point where the Berkeley regents were choosing the license for their UNIX code. Our time traveler is a very persuasive man, and convinces several regents that they should use the GPL. Most of the others receive a cup of coffee spiked with Richard Stallman’s messenger RNA. The last holdout is distracted on the morning of the final vote by the half dozen crazed ferrets that somehow got in his shower. BSD UNIX receives the GPL.

    Consider how the GPL would have been received in the corporate boardroom of the 1980s. BSD UNIX never would have been used.

    Organizations frequently build software features such that they work. That’s very different from “they work correctly,” but that’s what we would have gotten.

    Imagine SunOS shipping without BSD’s vi; you’d have ed as the default editor. Imagine Digital UNIX with a “good enough” virtual memory system. SVR4 wouldn’t have had long filenames, or job control. The standard Unix File System is a BSD invention; SVR4 would have shipped with the S51K filesystem. A power failure would have meant hours of work using tools like fsdb and clri. If you’ve never heard of fsdb and clri, you don’t want to know about them. Trust me on this one.

    If all that didn’t make you flinch: imagine the toll in sheer human suffering if Microsoft hadn’t used the BSD TCP/IP stack.

    If Windows was the only competition faced by Linux, Linux would not have come so far. Today, the Linux and BSD camps work every day to outpace each other. We don’t know where that race will end. We do know that the software at the end of it will work.

    Today, the attitude towards open source software is quite different than it was in the 1980s. The corporate world accepts open source, and even the GPL. Many will not release their code under the GPL, but they contribute many improvements back to to the BSD world and from there to everyone.

    We can never be sure what the world will think tomorrow, or next year. After all, forty years ago everyone knew you needed source code to do anything useful. Twenty years ago, source code was irrelevant for many organizations. We’ve returned to the open source world, but the wheel might just keep turning through open source and back the other way.

    Think it can’t happen? Lots of people would like to make Vernor Vinge’s Ubiquitous Law Enforcement real. Many of them have a lot of money. There are alternative scenarios that would crush or marginalize open source operating systems, much as car tinkerers are a fringe group today.

    In 2020, or 2030, we might hear the words “Anyone remember the open source fad?” I don’t think it will happen, but history is littered with the rotting corpses of invulnerable juggernauts.

    Even if the open source movement collapses, BSD-licensed code will still be used. The open-source BSD groups will have support from people smart enough to recognize the benefits of open source, but who are hamstrung by organizational policy or legal problems.

    Average, everyday people, whose only interest in a computer is from 9 to 5, Monday through Friday, benefit. Imagine if your code could, every day, save one thousand people five minutes of annoyance, frustration, or downright hysteria, without them having to know or do anything. BSD-licensed code saves more than that, at no further cost to anyone.

    The BSD license is not about open source. Open source is simply a tool to get better software. It’s not about ideology. It’s about correct, stable, and above all, better software, not just for computer-literate individuals but for everyone. It’s about giving users peace of mind, and letting developers do original work.

    BSD is about making the world a better place. For everyone.

    updated 2014-01-31: corrected some GPLv3 terms

    Ansible and PF, plus NTP

    It seems that ntpd has turned into the latest DDOS amplifier. I run a lot of servers, and most of them use the standard ntp client. I need to verify that none of my servers can be used for DDOS amplification. To do this, I need to give all the clients a standard NTP configuration, pointing at my personal NTP servers.

    While my internal addresses need access to the port 123 on my servers, the public doesn’t. And I occasionally add internal addresses. Automating PF and NTP configuration via Ansible will simplify my life in the future.

    I’ve used Ansible templates to configure services before, but packet filters are a little different. Packet filtering rules involve lots of information about the local host, such as interface names and the various system roles. It’s entirely possible to write an Ansible template that expresses your PF ruleset, it just took a little work.

    First, I define an Ansible group for the NTP servers (as well as other server duties). The time servers run FreeBSD 9.2.

    Here’s the playbook, with added NTP.

    ---
    - hosts: ntp-servers
      user: ansible
      sudo: yes
    
      tasks:
    
      - name: enable ntpdate
        action: command sysrc ntpdate_enable=yes
      - name: enable ntpdate server
        action: command sysrc ntpdate_hosts=pool.ntp.org
      - name: enable ntpd
        action: command sysrc ntpd_enable=yes
    
      - name: copy ntp.server.conf to servers
        action: copy src=/home/ansible/freebsd/etc/ntp.server.conf
          dest=/etc/ntp.conf owner=root group=wheel mode=0644
        notify:
          - restart ntpd
    
      - include: tasks/pf-compile.yml
    
      handlers:
    
        - include: handlers/restarts.yml

    Simple enough, no?

    Except there’s nothing in here about the packet filter. Or restarting ntp.

    These functions are pretty common, so I’ve moved them to separate files. I might need to rebuild the packet filter rules for any number of playbooks, after all.

    The file tasks/pf-compile.yml looks like this.

    ---
    #build pf.conf from template
    
      - name: configure firewall
        template: src=/home/ansible/freebsd/etc/pf.conf.j2
          dest=/etc/pf.conf owner=0 group=0 mode=0444
          validate='/sbin/pfctl -nf %s'
        notify:
          - reload pf

    This task uses a jinja2 template to build a pf.conf specifically for this host, copies it to the host, validates its syntax, puts it in place, and triggers a PF reload. Always validate your files before deploying them. Ansible doesn’t prevent mistakes, but rather allows you to deploy mistakes faster than ever.

    Similarly, I’ve split the “restart services” handlers off into the file handlers/restarts.yml. Here are the relevant bits.

    ---
    #restart assorted services
    
      - name: restart ntpd
        service: name=ntpd state=restarted
    
      - name: reload pf
        action: shell /sbin/pfctl -f /etc/pf.conf

    So, where is this firewall template? That’s probably what dragged you here.

    #{{ ansible_managed }}
    #$Id: pf.conf.j2,v 1.2 2014/01/16 16:10:54 mwlucas Exp $
    
    ext_if="{{ ansible_default_ipv4.device }}"
    
    include "/etc/pf.mgmt.conf"
    include "/etc/pf.ournets.conf"
    
    set block-policy return
    set loginterface $ext_if
    set skip on lo0
    
    scrub in all
    block in all
    
    pass in on $ext_if proto icmp all
    pass in on $ext_if proto icmp6 all
    
    #this host may initiate anything
    pass out on $ext_if from any to any
    
    #mgmt networks can talk to this host on any service
    pass in on $ext_if from  to any
    
    #Allowed services, in port order
    
    {% if inventory_hostname in groups['dns'] %}
    #DNS access
    pass in on $ext_if proto {tcp, udp} from any to any port 53
    {% endif %}
    
    {% if inventory_hostname in groups['tftpd'] %}
    #allow tftp from the world
    pass in on $ext_if proto udp from any to any port 69
    {% endif %}
    
    {% if inventory_hostname in groups['ntp-servers'] %}
    #allow time from our networks
    pass in on $ext_if proto udp from  to any port 123
    {% endif %}
    
    #end of services

    The first bit of new (to me) trickery in this is getting the interface name. I use an Ansible-provided variable for this. Get the complete list of Ansible-provided variables for a host by running ansible -m setup hostname. The variable ansible_default_ipv4.device contains the network interface name. (If your host has multiple network-facing interfaces, you’ll need to modify this.)

    This PF ruleset pulls in two external files, one containing a list of management addresses and one containing the complete list of my internal networks.

    I allow access from my management networks, allow ICMP, default block, all the routine packet filter stuff. The next interesting bit is the allowed services. I check for the host’s presence in a group, and if it’s there’ I add a rule to permit the access that protocol needs.

    One detail that gave me trouble made me use inventory_hostname rather than ansible_fqdn or ansible_hostname to check group membership. I manage systems in several domains, and many of them have one name in our management systems and another in DNS. I put machines in Ansible by their fully qualified domain name. Using ansible_fqdn to get a hostname returns the hostname given in reverse DNS. inventory_hostname returns the hostname as it appears in the hosts file. If ansible_fqdn doesn’t match the hostname in the hosts file, the group comparison fails. Using inventory_hostname gave me one consistent set of hostnames for comparisons.

    So now I can easily deploy a secure NTP configuration to my servers. When I have to deploy some other service that requires updating the packet filter, I can include the same task file. And the handlers are now similarly reusable.

    Configuring the clients across several different operating systems will probably require Ansible roles, however. I’d best get on that next…

    Wanted: a VAX

    No, not for me. If I bring another piece of obsolete hardware into this house, it can have my chair because I’ll be sitting out at the curb.

    The OpenBSD Project builds all their packages on native hardware. Yes, it might take a month to build a complete package set on some of their platforms, but that’s okay.

    Their VAX recently died. They need a new one.

    I know that some of you have a VAX in storage, that you’ve been reluctant to surrender because it’s a cool toy. Now you can send it to a good home. You can tell everyone that you, personally, saved OpenBSD’s VAX support.

    How often can your pack rat tendencies make you a hero? This is your chance. Take it.

    The Desktop of Doom

    I’m sick of scrounging hardware for writing books. I’m sick of waiting for things to compile, managing disk space, and running out of memory. I finally got so sick of it that I decided to invest some serious cash in a research machine, in the hope that I wouldn’t need to hunt hardware piecemeal for the next five to six years.

    I solicited hardware advice from my modest horde of Twitter followers. After diving through realms of documentation, I came to realize that I didn’t want to build my own computer. I rarely configure, buy, or even touch hardware these days unless something goes unspeakably wrong, and there’s enough options out there that were I to attempt to do so, I’d waste both money and time. I certainly could figure it out, but that’s not where my strength is and the expertise I would acquire would be largely wasted outside this one task.

    I wanted eight hard drives, so that when the time is right I can write about assorted ZFS configurations. I wanted some flash disk on top of that. I wanted enough oomph to run umpteen virtual machines simultaneously. And I didn’t want to spend a whole lot.

    After wandering around the Internet, using assorted online system configuration tools to develop a rough idea of price, I contacted iX Systems and bought a system.

    I now have a shiny new desktop system that can “portsnap fetch extract” in two minutes. Because SATA3 is pretty sweet. My only problem is that the portable version of cwm that @blooouuup found for me is really slow & clunky when resizing windows, but I’m sure if I whine at the right people someone will fix it for me. (Now to figure out who the right people are.)

    For all those who asked me to share my shopping results: here’s a FreeBSD 11 dmesg. Thanks to bhyve, I can now run whatever I need. It lives under my desk, and is surprisingly quiet when I’m not building x11/xorg in 30 minutes or less. With WITNESS enabled.

    iX Systems might not advertise desktop systems, but they’ll do a nice one for you. Even if yours isn’t as awesome as mine.


    Copyright (c) 1992-2013 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 11.0-CURRENT #0 r260064: Mon Dec 30 16:28:06 UTC 2013
    root@grind.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
    FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
    WARNING: WITNESS option enabled, expect reduced performance.
    CPU: Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz (3700.08-MHz K8-class CPU)
    Origin = "GenuineIntel" Id = 0x306e4 Family = 0x6 Model = 0x3e Stepping = 4
    Features=0xbfebfbff
    Features2=0x7fbee3ff
    AMD Features=0x2c100800
    AMD Features2=0x1
    Standard Extended Features=0x281
    TSC: P-state invariant, performance statistics
    real memory = 34368126976 (32776 MB)
    avail memory = 33227055104 (31687 MB)
    Event timer "LAPIC" quality 600
    ACPI APIC Table: < >
    FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
    FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads
    cpu0 (BSP): APIC ID: 0
    cpu1 (AP): APIC ID: 1
    cpu2 (AP): APIC ID: 2
    cpu3 (AP): APIC ID: 3
    cpu4 (AP): APIC ID: 4
    cpu5 (AP): APIC ID: 5
    cpu6 (AP): APIC ID: 6
    cpu7 (AP): APIC ID: 7
    ioapic0 irqs 0-23 on motherboard
    ioapic1 irqs 24-47 on motherboard
    kbd1 at kbdmux0
    random: initialized
    acpi0: on motherboard
    acpi0: Power Button (fixed)
    acpi0: reservation of 0, a0000 (3) failed
    cpu0: on acpi0
    cpu1: on acpi0
    cpu2: on acpi0
    cpu3: on acpi0
    cpu4: on acpi0
    cpu5: on acpi0
    cpu6: on acpi0
    cpu7: on acpi0
    attimer0: port 0x40-0x43 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    atrtc0: port 0x70-0x71 irq 8 on acpi0
    Event timer "RTC" frequency 32768 Hz quality 0
    hpet0: iomem 0xfed00000-0xfed003ff on acpi0
    Timecounter "HPET" frequency 14318180 Hz quality 950
    Event timer "HPET" frequency 14318180 Hz quality 550
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
    pcib0: port 0xcf8-0xcff on acpi0
    pci0: on pcib0
    pcib1: irq 27 at device 1.0 on pci0
    pci1: on pcib1
    pcib2: irq 33 at device 2.0 on pci0
    pci2: on pcib2
    pcib3: irq 33 at device 2.2 on pci0
    pci3: on pcib3
    pcib4: irq 43 at device 3.0 on pci0
    pci4: on pcib4
    pcib5: irq 43 at device 3.1 on pci0
    pci5: on pcib5
    pcib6: irq 43 at device 3.2 on pci0
    pci6: on pcib6
    pci0: at device 4.0 (no driver attached)
    pci0: at device 4.1 (no driver attached)
    pci0: at device 4.2 (no driver attached)
    pci0: at device 4.3 (no driver attached)
    pci0: at device 4.4 (no driver attached)
    pci0: at device 4.5 (no driver attached)
    pci0: at device 4.6 (no driver attached)
    pci0: at device 4.7 (no driver attached)
    pci0: at device 5.0 (no driver attached)
    pci0: at device 5.2 (no driver attached)
    pcib7: irq 16 at device 17.0 on pci0
    pci7: on pcib7
    isci0: port 0xe000-0xe0ff mem 0xfa47c000-0xfa47ffff,0xfa000000-0xfa3fffff irq 16 at device 0.0 on pci7
    pci0: at device 22.0 (no driver attached)
    pci0: at device 22.1 (no driver attached)
    ehci0: mem 0xfbb23000-0xfbb233ff irq 16 at device 26.0 on pci0
    usbus0: EHCI version 1.0
    usbus0 on ehci0
    pcib8: irq 17 at device 28.0 on pci0
    pci8: on pcib8
    pcib9: irq 18 at device 28.6 on pci0
    pci9: on pcib9
    em0: port 0xd000-0xd01f mem 0xfba00000-0xfba1ffff,0xfba20000-0xfba23fff irq 18 at device 0.0 on pci9
    em0: Using MSIX interrupts with 3 vectors
    em0: Ethernet address: 00:25:90:db:d5:94
    pcib10: irq 19 at device 28.7 on pci0
    pci10: on pcib10
    em1: port 0xc000-0xc01f mem 0xfb900000-0xfb91ffff,0xfb920000-0xfb923fff irq 19 at device 0.0 on pci10
    em1: Using MSIX interrupts with 3 vectors
    em1: Ethernet address: 00:25:90:db:d5:95
    ehci1: mem 0xfbb22000-0xfbb223ff irq 23 at device 29.0 on pci0
    usbus1: EHCI version 1.0
    usbus1 on ehci1
    pcib11: at device 30.0 on pci0
    pci11: on pcib11
    vgapci0: mem 0xf9000000-0xf9ffffff,0xfb800000-0xfb803fff,0xfb000000-0xfb7fffff irq 16 at device 4.0 on pci11
    vgapci0: Boot video device
    isab0: at device 31.0 on pci0
    isa0: on isab0
    ahci0: port 0xf050-0xf057,0xf040-0xf043,0xf030-0xf037,0xf020-0xf023,0xf000-0xf01f mem 0xfbb21000-0xfbb217ff irq 18 at device 31.2 on pci0
    ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
    ahcich0: at channel 0 on ahci0
    ahcich1: at channel 1 on ahci0
    ahcich2: at channel 2 on ahci0
    ahcich3: at channel 3 on ahci0
    ahcich4: at channel 4 on ahci0
    ahcich5: at channel 5 on ahci0
    ahciem0: on ahci0
    pci0: at device 31.3 (no driver attached)
    pci0: at device 31.6 (no driver attached)
    pcib12: on acpi0
    pci255: on pcib12
    pci255: at device 8.0 (no driver attached)
    pci255: at device 9.0 (no driver attached)
    pci255: at device 10.0 (no driver attached)
    pci255: at device 10.1 (no driver attached)
    pci255: at device 10.2 (no driver attached)
    pci255: at device 10.3 (no driver attached)
    pci255: at device 11.0 (no driver attached)
    pci255: at device 11.3 (no driver attached)
    pci255: at device 12.0 (no driver attached)
    pci255: at device 12.1 (no driver attached)
    pci255: at device 13.0 (no driver attached)
    pci255: at device 13.1 (no driver attached)
    pci255: at device 14.0 (no driver attached)
    pci255: at device 14.1 (no driver attached)
    pci255: at device 15.0 (no driver attached)
    pci255: at device 15.1 (no driver attached)
    pci255: at device 15.2 (no driver attached)
    pci255: at device 15.3 (no driver attached)
    pci255: at device 15.4 (no driver attached)
    pci255: at device 15.5 (no driver attached)
    pci255: at device 16.0 (no driver attached)
    pci255: at device 16.1 (no driver attached)
    pci255: at device 16.2 (no driver attached)
    pci255: at device 16.3 (no driver attached)
    pci255: at device 16.4 (no driver attached)
    pci255: at device 16.5 (no driver attached)
    pci255: at device 16.6 (no driver attached)
    pci255: at device 16.7 (no driver attached)
    pci255: at device 19.0 (no driver attached)
    pci255: at device 19.1 (no driver attached)
    pci255: at device 19.4 (no driver attached)
    pci255: at device 19.5 (no driver attached)
    pci255: at device 22.0 (no driver attached)
    pci255: at device 22.1 (no driver attached)
    pci255: at device 22.2 (no driver attached)
    acpi_button0: on acpi0
    uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
    uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
    uart2: <16550 or compatible> port 0x3e8-0x3ef irq 10 on acpi0
    orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0
    sc0: at flags 0x100 on isa0
    sc0: VGA <16 virtual consoles, flags=0x300>
    vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
    ppc0: cannot reserve I/O port range
    est0: on cpu0
    p4tcc0: on cpu0
    est1: on cpu1
    p4tcc1: on cpu1
    est2: on cpu2
    p4tcc2: on cpu2
    est3: on cpu3
    p4tcc3: on cpu3
    est4: on cpu4
    p4tcc4: on cpu4
    est5: on cpu5
    p4tcc5: on cpu5
    est6: on cpu6
    p4tcc6: on cpu6
    est7: on cpu7
    p4tcc7: on cpu7
    ZFS filesystem version: 5
    ZFS storage pool version: features support (5000)
    Timecounters tick every 1.000 msec
    random: unblocking device.
    usbus0: 480Mbps High Speed USB v2.0
    usbus1: 480Mbps High Speed USB v2.0
    ugen0.1: at usbus0
    uhub0: on usbus0
    ugen1.1: at usbus1
    uhub1: on usbus1
    uhub1: 2 ports with 2 removable, self powered
    uhub0: 2 ports with 2 removable, self powered
    ses0 at ahciem0 bus 0 scbus7 target 0 lun 0
    ses0: SEMB S-E-S 2.00 device
    ses0: SEMB SES Device
    da2 at isci0 bus 0 scbus0 target 2 lun 0
    da2: Fixed Direct Access SCSI-5 device
    da2: Serial Number WD-WCAW36477223
    da2: 300.000MB/s transfers
    da2: Command Queueing enabled
    da2: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
    da0 at isci0 bus 0 scbus0 target 0 lun 0
    da0: Fixed Direct Access SCSI-5 device
    da0: Serial Number WD-WCAW36478143
    da0: 300.000MB/s transfers
    da0: Command Queueing enabled
    da0: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
    da3 at isci0 bus 0 scbus0 target 3 lun 0
    da3: Fixed Direct Access SCSI-5 device
    da3: Serial Number WD-WCAW36477062
    da3: 300.000MB/s transfers
    da3: Command Queueing enabled
    da3: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
    da1 at isci0 bus 0 scbus0 target 1 lun 0
    da1: Fixed Direct Access SCSI-5 device
    da1: Serial Number WD-WCAW36498185
    da1: 300.000MB/s transfers
    da1: Command Queueing enabled
    da1: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
    ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
    ada0: ATA-8 SATA 3.x device
    ada0: Serial Number WD-WCAW36478129
    ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
    ada0: Command Queueing enabled
    ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
    ada0: Previously was known as ad4
    ada1 at ahcich1 bus 0 scbus2 target 0 lun 0
    ada1: ATA-8 SATA 3.x device
    ada1: Serial Number WD-WCAW36477352
    ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
    ada1: Command Queueing enabled
    ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
    ada1: Previously was known as ad6
    ada2 at ahcich2 bus 0 scbus3 target 0 lun 0
    ada2: ATA-8 SATA 3.x device
    ada2: Serial Number WD-WCAW36477139
    ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
    ada2: Command Queueing enabled
    ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
    ada2: Previously was known as ad8
    ada3 at ahcich3 bus 0 scbus4 target 0 lun 0
    ada3: ATA-8 SATA 3.x device
    ada3: Serial Number WD-WCAW36477141
    ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
    ada3: Command Queueing enabled
    ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
    ada3: Previously was known as ad10
    ada4 at ahcich4 bus 0 scbus5 target 0 lun 0
    ada4: ATA-8 SATA 2.x device
    ada4: Serial Number 20131111AA300000000E
    ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
    ada4: Command Queueing enabled
    ada4: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C)
    ada4: Previously was known as ad12
    ada5 at ahcich5 bus 0 scbus6 target 0 lun 0
    ada5: ATA-8 SATA 2.x device
    ada5: Serial Number 20131111AA3000000007
    ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
    ada5: Command Queueing enabled
    ada5: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C)
    ada5: Previously was known as ad14
    ugen0.2: at usbus0
    uhub2: on usbus0
    ugen1.2: at usbus1
    uhub3: on usbus1
    Netvsc initializing... done!
    SMP: AP CPU #1 Launched!
    SMP: AP CPU #3 Launched!
    SMP: AP CPU #4 Launched!
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #6 Launched!
    SMP: AP CPU #7 Launched!
    SMP: AP CPU #5 Launched!
    Timecounter "TSC-low" frequency 1850039224 Hz quality 1000
    WARNING: WITNESS option enabled, expect reduced performance.
    GEOM_RAID: Intel-ee147409: Array Intel-ee147409 created.
    GEOM_RAID: Intel-ee147409: Disk ada4 state changed from NONE to ACTIVE.
    GEOM_RAID: Intel-ee147409: Subdisk Volume0:0-ada4 state changed from NONE to ACTIVE.
    GEOM_RAID: Intel-ee147409: Disk ada5 state changed from NONE to ACTIVE.
    GEOM_RAID: Intel-ee147409: Subdisk Volume0:1-ada5 state changed from NONE to ACTIVE.
    GEOM_RAID: Intel-ee147409: Array started.
    GEOM_RAID: Intel-ee147409: Volume Volume0 state changed from STARTING to OPTIMAL.
    GEOM_RAID: Intel-ee147409: Provider raid/r0 for volume Volume0 created.

    Jailing FreeBSD 4 on FreeBSD 10

    We have an in-house application that was written for FreeBSD 4 and antediluvian versions of PHP, Perl, OpenSSL, and so forth. Most of the features have migrated into other applications, but a few critical functions remain.

    An old operating system isn’t sufficiently bad, though. The hardware terrifies me. Not only is it over a decade old, it’s repurposed desktop hardware.

    Virtualize it? Maybe. But device drivers have changed over the intervening decade, and a ten-year-old de(4) or fxp(4) driver works poorly on any of my virtualization systems. Virtio is right out.

    Port it to a current OS, PHP, and perl? That would be a painful prospect if I knew what I was doing. I’m a sysadmin, not a programmer. I have no bloody clue what I’m doing.

    But theoretically, FreeBSD 4 systems should run almost unchanged in a jail on FreeBSD 10. Can they? Let’s find out!

    First I tarred up the entire 4.10 system, except for /proc. (Yes, FreeBSD 4.10 used /proc. Those were the days, eh?) I did no preparatory cleaning, and even included port work directories, /usr/obj, /var/tmp, and so on, as I have NO idea what I might need. Yes, I’m sure I can find a PHP 5.0.whatever tarball out on the Internet, but that would involve work.

    Create my jail directory, and untar the copied server

    # mkdir /var/jail/oldserver
    # cd oldserver
    # tar -xvpf $HOME/oldserver.tgz

    Be sure to use the -p flag, to preserve permissions.

    Now to edit some configuration files, to change this system from hardware to jail.

  • Check rc.conf. Remove functions handled by the host server: firewalls, timekeeping, SSH, and so on. This jailed host only needs the functions that directly support the application (the Apache web server and the associated MySQL database).
  • Check /etc/fstab for anything you need to retain, like NFS filesystems or /proc. Comment out everything else.
  • Remove the old /dev. You cannot use FreeBSD 4 device nodes on a FreeBSD 10 host. (I spent an hour repeatedly bashing my head on the brick wall intensively testing these two sentences, so you can rest assured that it’s true.)

    Now on to the jail server. FreeBSD 9 and above has a jail-specific configuration file, /etc/jail.conf. I define some basic characteristics here.

    While that’s running, do some basic jail setup. I would normally recommend ezjail, but this is a fairly special case: no ZFS, the server will never be upgraded, and I specifically want a very minimal system. Also, ezjail doesn’t seem to have been updated for the Jail New World Order. So I’ll configure this the hard way, which isn’t terribly hard.

    FreeBSD 9 and above has a jail-specific configuration file, /etc/jail.conf. Here’s a configuration for my old server.

    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";

    oldserver {
    path = /var/jail/oldserver;
    host.hostname = oldserver.mwlucas.org;
    ip4.addr = 10.0.16.31;
    interface = vtnet0;
    };

    Now for the real test:

    # service jail start oldserver
    Starting jails: oldserver
    #

    Is it that easy?

    Of course not. Several processes didn’t start. I had to edit some configuration files to account for the change of IP address. That’s pretty minor, though.

    To log into the jail, get the jail ID and run a shell prompt in that server.

    host# jls
    JID IP Address Hostname Path
    7 10.0.16.31 oldserver.mwlucas.org /var/toolkit
    host# jexec 7 /bin/tcsh
    oldserver#

    The advantage to hosting this ancient system as a jail is that I can use the host operating system to control access to the legacy application. I don’t have to use the ancient IPF included on FreeBSD 4.10; I protect this host with PF from FreeBSD 10. (Admittedly, there’s newer PF out there, but even the older version in FreeBSD 10 is better than IPF.) FreeBSD 4.10 includes OpenSSH 3.5, which has roughly the same security as Windows XP. To log into the old server now, you must authenticate to the up-to-date FreeBSD 10 system and get a jail command prompt.

    The disadvantage to this system is that you can’t run ps(1) inside the jail. The ps command reads kernel data structures. While the FreeBSD 10 kernel includes FreeBSD 4 compatibility, that compatibility doesn’t extend to ps(1) and similar commands. You must use ps(1) from the jail host.

    That’s an acceptable trade-off, if it means I don’t have to touch actual PHP code. That stuff has cooties.