Running Ancient Rsync

Another “write it down so I don’t forget what I did” post.

Some of the systems I’m responsible for are file storage machines, running rsync 3.0 or 3.1 as a daemon. Every hour, an ancient Solaris machine sends files to it using rsync 2.3.1. The billing team uses these files to create bills.

Thursday, I rebooted the machine. And the rsync stopped working with:

rsyncd[3582]: rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsyncd[3582]: rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.0]

The rsyncd server hadn’t changed. No security patches, no package updates, no nothing.

We cannot change the software on the Solaris machine. It’s attached to a multimillion-dollar telco switch, and editing the software on it would invalidate the warranty. The whole point of buying a multimillion-dollar telco switch is so you get the warranty. If something goes wrong, a team of vendor experts descends on your facility with enough spare parts to rebuild the switch from the ground up. (Telephony is nothing like IT. Really.) I cannot use SSH to transfer the files. I do not administer this machine–actually, I don’t want to administer this machine. I’m so unfamiliar with warranties on operating systems that I would probably void it by copying my SSH public key to it or something.

The Solaris box is running rsync 2.3.1, which runs rsync protocol version 20. My systems use newer rsync, running protocol version 30 or 31.

Rsyncd isn’t easily debuggable. Packet analysis showed messages about protocol errors. The rsync FAQ has a whole bunch of troubleshooting suggestions. None of them worked. I ran rsync under truss and strace and painstakingly read system calls. I eventually sacrificed a small helpless creature in accordance with ancient forbidden rites under last weekend’s full moon.

After a few days of running through a backup system (an old but not quite ancient OpenSolaris box), I absolutely had to get this working. So: protocol errors? Let’s try an older rsync.

Rsync 2.9? Same problem. I saw myself progressively working my way through building older versions, solving weird problems one by one, and eventually finding something old enough to work. This is not how I wanted to spend my week. Given how well running FreeBSD 4 in a FreeBSD 10 jail works, I tried something similar.

The host ftp-archive.freebsd.org host releases of every FreeBSD version, including packages. FreeBSD 10 includes compatibility with FreeBSD back to version 4. I installed the compatibility libraries from /usr/ports/misc/compat4.

The oldest FreeBSD 4 rsync package I could find was 2.4.6, from FreeBSD 4.1.1. Original FreeBSD packages were just zipped tar files. I extracted the files and checked that the binary could find all its libraries.

# ldd rsync
rsync:
libc.so.4 => /usr/local/lib32/compat/libc.so.4 (0x2808a000)

If this was more complicated software, with more libraries, I’d have to track down the missing ones. Rsync is very straightforward, however.

I shut down the old rsync daemon and fired up the old one.

It worked.

I still want to know how a reboot broke this. I’m assuming that something changed and that I lack the sysadmin chops to identify it. It’s not the rsync binary, or libc; both have date stamps several months old.

I don’t recommend this, as older rsync has all kinds of security problems. These particular hosts are behind several layers of firewalls. If an intruder gets this far, I’m basically doomed anyway.

So: if you’re very very stuck, and the clock has run out, using really old software is an option. But it still makes my skin crawl.

Trying poo-DRE-eh — uh, poudriere

This is my poudriere tutorial. There are many like it. But this one is mine. I built mine with resources like the BSDNow tutorial and the FreeBSD Forums tutorial. While all poudriere tutorials are inadequate, mine is inadequate in new and exciting ways. I’m writing it for my benefit, but what the heck, might as well post it here. (If you read this and say “I learned nothing new,” well, I warned you.)

Your package building system must run the newest version of FreeBSD you want to support. I have 8, 9, and 10 in production, so my package builder needs to be FreeBSD 10 or newer. I’m using FreeBSD 11, because I’m running my package builder on my desktop. I upgraded this machine to the latest -current and updated all my packages and ports.

Poudriere works on either UFS or ZFS partitions. I have copious disk, so I’ll dedicating two of them to poudriere. (This means I’ll have to blow away my install later when I start experimenting with disks, hence this blog posting.) I use disks ada2 and ada3.

First, eradicate anything already on those disks.

# gpart destroy -F ada3
ada3 destroyed
# gpart destroy -F ada2
ada2 destroyed
#

Now I can create new GPT partitions. Each disk needs one partition for ZFS, covering the whole disk. You’ll find lots of discussion about partitioning disks with 512-byte sectors versus those with 4KB sectors, and the need for carefully aligning your disk partitions with the underlying sector size. The easy way around this is to duck the whole issue and assume 4KB sectors, use a null GEOM layer to align everything to the disk’s expectations, and ZFS the null layer. (If you KNOW the sector size of your disk, you can simplify the below, but remember, disks lie about their sector size just like they lie about geometry. It’s safest to gnop all the things.)

# gpart create -s gpt ada2
# gpart create -s gpt ada3
# gpart add -a 4k -t freebsd-zfs -l disk2 ada2
# gpart add -a 4k -t freebsd-zfs -l disk3 ada3
# gnop create -S 4096 /dev/gpt/disk2
# gnop create -S 4096 /dev/gpt/disk3
# zpool create poudriere mirror /dev/gpt/disk2.nop /dev/gpt/disk3.nop

I now have a filesystem to build packages on. Let’s get me a key to sign them with.

# mkdir -p /usr/local/etc/ssl/keys
# mkdir -p /usr/local/etc/ssl/certs
# cd /usr/local/etc/ssl
# openssl genrsa -out keys/mwlucas.key 4096
# openssl rsa -in keys/mwlucas.key -pubout > certs/mwlucas.cert

Installed the latest poudriere from /usr/ports/ports-mgmt/poudriere-devel. While the configuration file is lengthy, you don’t need to set many options.

ZPOOL=poudriere
FREEBSD_HOST=ftp://ftp.freebsd.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/poudriere
POUDRIERE_DATA=${BASEFS}/data
USE_PORTLINT=no
USE_TMPFS=yes
DISTFILES_CACHE=/usr/ports/distfiles
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/mwlucas.key
URL_BASE=http://storm.michaelwlucas.com/

I set the CHECK_CHANGED options because when I update my ports tree, I want to know about changed options before I build and deploy my packages. I set the URL_BASE so I can view the build logs on my web server.

The build process uses its own make.conf file, /usr/local/etc/poudriere.d/make.conf, where I set some very basic things. The only mandatory setting is WITH_PKGNG. You could also have a separate make.conf for each individual jail, but I want all of my packages consistent, so I only use the central file.

WITH_PKGNG="yes"
WITHOUT_X11="yes"
WITHOUT_HAL="yes"
WITHOUT_NLS="yes"

Get a ports tree just for poudriere. I could use the ports tree on the package builder, but it’s possible that ports on the builder might differ from what I want for the packages. It’s best to keep everything tidy.

# poudriere ports -c
====>> Creating default fs... done
====>> Extracting portstree "default"...
Looking up portsnap.FreeBSD.org mirrors... 7 mirrors found.
Fetching public key from your-org.portsnap.freebsd.org... done.
Fetching snapshot tag from your-org.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
...

Now create a jail to build packages in.

At first blush, you might give the jail any random name. Using the same jail standard as the FreeBSD package builder eases deployment, however. The official naming standard combines operating system, CPU architecture, operating system version, and word size, like so. For example, 32-bit FreeBSD 10 on Intel-type processors is freebsd:10:x86:32, while the 64-bit version is freebsd:10:x86:64. (See pkg-repository for details.)

FreeBSD’s official package builds occur on the oldest supported version of a release. Packages for 9-stable are built on 9.1, so I do:

# poudriere jail -c -j freebsd:9:x86:32 -v 9.1-RELEASE -a i386

Then create jails for the other two releases I support.

# poudriere jail -c -j freebsd:9:x86:32 -v 9.1-RELEASE -a amd64
# poudriere jail -c -j freebsd:10:x86:64 -v 10.0-RELEASE -a amd64

I only run the amd64 version of FreeBSD 10, because I don’t want to build i386 packages forever.

This takes a little while, so start it and walk away. I copied these lines into a shell script and went to lunch.

Note that these are not actual jails. They will not show up in jls(8). Technically, they’re chroots. (I’m not sure why poudriere calls them jails – maybe they were originally such but the need for a full jail went away, or they’re intended to become full jails later on. I’m sure there’s a perfectly sensible reason, however.) You can list your package-building jails only via poudriere.

# poudriere jail -l
JAILNAME VERSION ARCH METHOD PATH
freebsd:9:x86:64 9.1-RELEASE amd64 ftp /poudriere/jails/freebsd:9:x86:64
freebsd:9:x86:32 9.1-RELEASE i386 ftp /poudriere/jails/freebsd:9:x86:32
freebsd:10:x86:64 10.0-RELEASE amd64 ftp /poudriere/jails/freebsd:10:x86:64

Before building any packages, I want to update all the jails to the latest version. As I’ll need to do this before every package build, I script it. I also add the command to update poudriere’s ports tree.

#!/bin/sh

#update ports tree
poudriere ports -p default -u

#Update known builder jails to latest version
poudriere jail -u -j freebsd:9:x86:32
poudriere jail -u -j freebsd:9:x86:64
poudriere jail -u -j freebsd:10:x86:64

I must run this script every time I update my packages.

Now to determine which packages I want to build. In my case, I want to build only packages that I can’t get from official FreeBSD sources. This means things like freeradius and Apache with LDAP support and PHP with Apache support. Simple enough. I created /usr/local/etc/poudriere.d/pkglist.txt containing:

#web services - need LDAP & Apache PHP module
www/apache22
lang/php5
#network - need LDAP & SMB
net/freeradius2
net/openldap24-server

Now the fun part: setting the build options for these ports.

# poudriere options -cf pkglist.txt

This takes you into a recursive make config for all of your selected packages. If you know exactly which ports you must configure to get a properly built package, you could specify ports by name. Unfortunately, I always forget to add LDAP to some dependency, so I walk through all the configurations adding my various options.

You can now build your packages. I want to update all of my packages simultaneously, so I wrote a trivial shell script to build packages.

#!/bin/sh

#Update known builder jails to latest version

poudriere bulk -f /usr/local/etc/poudriere.d/pkglist.txt -j freebsd:9:x86:32
poudriere bulk -f /usr/local/etc/poudriere.d/pkglist.txt -j freebsd:9:x86:64
poudriere bulk -f /usr/local/etc/poudriere.d/pkglist.txt -j freebsd:10:x86:64

Walk away. Or, if you prefer, set up a web server so you can see the build progress and deliver packages to clients. I used apache22 and created /usr/local/etc/apache22/Includes/poudriere.conf containing:

NameVirtualHost *:80


ServerAdmin mwlucas@minetworkservices.com
DocumentRoot /poudriere/data
ServerName storm.blackhelicopters.org


Options Indexes FollowSymLinks
AllowOverride AuthConfig
Order allow,deny
Allow from all

Yes, I could have hacked up httpd.conf to set DocumentRoot, but I prefer to leave package-created files alone if possible.

Let’s turn to the client while this builds. The client needs a repository configuration file and a copy of the build certificate. Here’s the configuration for my local repository, named mwlucas.

mwlucas: {
url: "http://storm.blackhelicopters.org/packages/${ABI}-default/",
mirror_type: "http",
signature_type: "pubkey",
pubkey: "/usr/local/etc/ssl/certs/mwlucas.cert",
fingerprints: "/usr/share/keys/pkg",
enabled: yes
}

By using the standard jail names, I was able to use the ${ABI} variable in my repository path. This saves me from needing to update my repository configuration every time I upgrade.

I copy my certificate to the directory specified in the pubkey option.

You should be able to browse to the URL given as your repository. If the URL doesn’t work, you misconfigured your web server.

Check your repository configuration with pkg -vv. You should see all configured repositories.

# pkg -vv
...
Repositories:
FreeBSD: {
url : "pkg+http://pkg.FreeBSD.org/freebsd:10:x86:64/latest",
enabled : yes,
mirror_type : "SRV",
signature_type : "FINGERPRINTS",
fingerprints : "/usr/share/keys/pkg"
}
mwlucas: {
url : "http://storm.blackhelicopters.org/packages/freebsd:10:x86:64-default/Latest/",
enabled : yes,
mirror_type : "HTTP",
signature_type : "PUBKEY",
fingerprints : "/usr/share/keys/pkg",
pubkey : "/usr/local/etc/ssl/certs/mwlucas.cert"
}

Two repositories? Good. But can you actually access the repository? A pkg update should pull down the digests for your new repo.

# pkg update
Updating repository catalogue
digests.txz 100% 1928 1.9KB/s 1.9KB/s 00:00
packagesite.txz 100% 10KB 10.0KB/s 10.0KB/s 00:00
Incremental update completed, 23 packages processed:
0 packages updated, 0 removed and 23 added.

A quick check will verify that the private repo has 23 packages.

Now for the weakest part of pkgng: actually using your repository for select packages. If I built everything, I’d just disable the FreeBSD repository. But I want to only build packages that differ from the default.

pkg searches for packages in each repository in order. At the moment, repositories appear in alphabetical order. I could rename my repository so that it appears before FreeBSD. pkg would search my repo for packages, and if they didn’t exist there check the FreeBSD repo.

This would be ideal. But alphabetical repository ordering is not guaranteed. The only way to ensure the packages install from the correct repository is to tell each individual FreeBSD server where it should get specific packages. This kind of sucks, but it’s still an improvement on pkg_add. (I hear that repository handling should improve in future versions of pkg.)

Use the -r flag to specify a repository with pkg. For example, let’s search my private repo for an Apache package.

# pkg search -r mwlucas apache
apache22-2.2.26

The package is there. Excellent. Install it.

# pkg install -r mwlucas apache22
Updating repository catalogue
The following 8 packages will be installed:

Installing expat: 2.1.0 [mwlucas]
Installing perl5: 5.16.3_7 [mwlucas]
Installing pcre: 8.34 [mwlucas]
Installing openldap-client: 2.4.38 [mwlucas]
Installing gdbm: 1.11 [mwlucas]
Installing db42: 4.2.52_5 [mwlucas]
Installing apr: 1.4.8.1.5.3 [mwlucas]
Installing apache22: 2.2.26 [mwlucas]

The installation will require 94 MB more space

19 MB to be downloaded

Proceed with installing packages [y/N]: y

Take careful note of this list of dependency packages installed from your repository.

Here you have to decide how you want to upgrade your server. All packages that you want to upgrade from your repo need to be marked as such. How many of these dependency packages must be upgraded via your repo? That’s a good question. If you took note of which packages you had to change the configuration of, way back in the beginning, you could label only those changed packages as requiring your repo. I didn’t do that. If this was a brand-new machine I would install packages from my repo first and mark everything installed as requiring my repo. I didn’t do that. Instead, I’m going to mark all packages required by apache22 as belonging to my repo, much like this.

# pkg annotate -Ay apache22 repository mwlucas
apache22-2.2.26: added annotation tagged: repository
# pkg annotate -Ay pcre repository mwlucas

Repeat this for each package installed by this package install.

The repository tag appears when you run pkg info on a specific package.

# pkg info pcre | grep repository
repository : mwlucas

pkg upgrade will now only use your repository for the specified package.

Is this long and complicated? Sure. But it beats the snot out of maintaining three separate package repositories under the old package system.

Installing FreeBSD 10 to ZFS with a script

Well, partially scripted, that is.

For installing large numbers of identical machines, proceed directly to the PC-BSD installer. It’s easy to configure, very reliable, and generally just rocks. If you’re accustomed to automatic installers like Kickstart, you’ll find the PC-BSD installer trivially easy.

I frequently have to install non-identical machines for special purposes, such as testing or unique file stores or EDI. Most of these are virtual machines. It seems that ZFS filesystems compress really really well, simplifying backing up the VMs.

And there, the FreeBSD 10 installer’s ZFS features don’t quite cut it. The installer lets you create a single large ZFS without leaving the GUI. I want a more complicated ZFS setup, based on the FreeBSD Root on ZFS wiki page. This process involves a whole lot of typing. I normally install servers when I’m too brain dead to do any real work, so I need to minimize the opportunity for errors.

Fortunately, you can script all the disk and ZFS setup. Here’s my script. If it looks familiar, well, it should: it’s ripped raw and bleeding from the wiki instructions and wrapped up with /bin/sh -x. (I use -x because I want to see how the script runs.)

#!/bin/sh -x

#Auto-divides a ZFS install.
#ZFS permissions stolen from
#https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/9.0-RELEASE

#edit:
#disk device name
#parameters for your zpool type
#your pool name
#swap space

#we're installing
sysctl kern.geom.debugflags=0x10

gpart destroy -F vtbd0
gpart create -s gpt vtbd0
gpart add -s 222 -a 4k -t freebsd-boot -l boot0 vtbd0

gpart add -s 1g -a 4k -t freebsd-swap -l swap0 vtbd0
gpart add -a 4k -t freebsd-zfs -l disk0 vtbd0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 vtbd0
gnop create -S 4096 /dev/gpt/disk0

kldload zfs
zpool create -f -o altroot=/mnt -O canmount=off -m none zroot /dev/gpt/disk0.nop

zfs set checksum=fletcher4 zroot
zfs set atime=off zroot

zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default
zfs create -o mountpoint=/tmp -o compression=lzjb -o setuid=off zroot/tmp
chmod 1777 /mnt/tmp

zfs create -o mountpoint=/usr zroot/usr
zfs create zroot/usr/local

zfs create -o mountpoint=/home -o setuid=off zroot/home
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages

zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src
zfs create zroot/usr/obj

zfs create -o mountpoint=/var zroot/var
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
chmod 1777 /mnt/var/tmp
zpool set bootfs=zroot/ROOT/default zroot

cat << EOF > /tmp/bsdinstall_etc/fstab
/dev/gpt/swap0 none swap sw 0 0
EOF

exit

How do I use this?

When the installer asks me how I want to partition the disk, I exit to the shell, configure the network, and get the script onto the target system.

# dhclient vtnet0
# cd /tmp
# fetch http://www-old.michaelwlucas.com/zfsinstall.sh
# chmod 755 zfsinstall.sh
# ./zfsinstall.sh

Sit back and watch it run. When the script finishes, exit from the shell and let the installer unpack the files. When you’re offered a shell to perform post-install configuration, take it and run the post-install ZFS setup commands.

# mount -t devfs devfs /dev
# echo 'zfs_enable="YES"' >> /etc/rc.conf
# echo 'zfs_load="YES"' >> /boot/loader.conf
# zfs set readonly=on zroot/var/empty

Reboot into your fine-grained ZFS filesystem installation.

To use this script yourself, you’ll need to check the disk device name and the type of zpool you want to create. But this will hopefully get you started.

I would really like to see the default FreeBSD installer create finer grained ZFS filesystems. I’m told that day is coming.

ifup-local on bridge members on CentOS

I run a bunch of CentOS 6 physical servers as QEMU virtualization devices. These hosts have two NICs, one for management and one for virtual machine bridges.

When you use Linux for virtualization, it’s important to increase the amount of memory for network transmit and receive buffers. You also need to disable GSO and TSO, to improve performance and to avoid gigabytes of kernel error messages every day. You can do this with ethtool(8). First, let’s check the existing ring sizes.

# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX: 16384
RX Mini: 0
RX Jumbo: 0
TX: 16384
Current hardware settings:
RX: 512
RX Mini: 0
RX Jumbo: 0
TX: 512

Similarly, use ethtool -k eth0 to check GSO and TSO settings.

The card is using much less memory than it can. When you have a bunch of virtual machines pouring data through the card, you want the card to work as efficiently as possible. Fixing this on a running system is easy enough:

# ethtool -G eth0 tx 16384 rx 16384
# ethtool -K eth0 gso off tso off

Repeat the process for eth1.

How do you make this happen automatically at boot? Adding the commands to /etc/rc.local isn’t reliable. By the time the system gets that much stuff running, the ethtool command might fail with a “Cannot allocate memory” error. If you try again it’ll probably work, but it’s not deterministic. And I’m against running a single command four times in rc.local in the hopes that one of them will work.

Enter /sbin/ifup-local. CentOS runs this script after bringing up an interface, with the interface name as an argument. The problem is, it doesn’t run this script on bridge member interfaces. We can adjust eth0 and br0 at boot just fine, but eth1 (the physical interface underlying br0) doesn’t get run.

You can’t run ethtool -G eth0 tx 16384 rx 16384 on br0. Interface br0 doesn’t have any transmit or receive rings. It’s a logical interface. You can disable TSO and GSO on br0, but that won’t disable it on eth1. You can’t wait to reconfigure eth1 in rc.local until the system is running, because increasing the memory doesn’t always work once the system is running full-out multiuser. And Red Hat says this is by design. Apparently network bridges on CentOS/Red Hat are supposed to perform poorly. That’s good to know.

So, what to do?

I adjust the eth1 ring size in ifup-local when bringing up br0, but before any processes send any traffic over the bridge. My /sbin/ifup-local looks like this:

#!/bin/bash

case "$1" in
eth0)
echo "Configuring eth0..."
/sbin/ethtool -G eth0 tx 16384 rx 16384
/sbin/ethtool -K eth0 gso off tso off
;;

br0)
echo "Configuring br0..."
/sbin/ethtool -G eth1 tx 16384 rx 16384
/sbin/ethtool -K eth1 gso off tso off
/sbin/ethtool -K br0 gso off tso off
;;

esac
exit 0

This appears to work consistently. Of course, the values for the NIC need to be set on a per-machine basis. I have Ansible do that work for me.

Hopefully, this will save someone else the pain I’ve been through trying to make this work…

New reviews

There’s been a few new reviews out lately. First, two from Grant Taylor, on Sudo Mastery and SSH Mastery. Thank you, Grant!

Yesterday, a review of Sudo Mastery appeared on Slashdot. I haven’t been reviewed on Slashdot since Absolute OpenBSD came out. No, not the second edition–the original, in 2003. So this is cool. Thank you, “Saint Aardvark.” (Yes, I can figure out his real name, but if he goes by that, who am I to argue?)

As a result of these reviews, I now simultaneously have the #1 and #4 best-seller slots in Amazon’s Unix category.

I really want to thank everyone who takes the time to review my books — or, indeed, any books. Reviews drive sales. Sales mean that authors can afford to write books instead of washing dishes at the Burger Hut (which is all that most of us are qualified for in the real world). If you enjoy a book, and want to thank the author, take a moment to do so publicly.

And now back to writing more books…

2 titles in Amazon's top 10

2013 Failures and 2014 Goals

I set goals for 2013. And I failed to meet them. I promised three short nonfiction books, Absolute OpenBSD 2nd edition, and a novel. You got AO2e and two short nonfiction books, DNSSEC Mastery and Sudo Mastery.

While setting goals is important, exploring why you fail to meet those goals is just as important. Driving factors behind these goals boil down to three things.

  • These were pretty ambitious goals
  • Traveled to EuroBSDCon in September
  • January’s emergency appendectomy
  • I knew this was ambitious beforehand, but decided to try for it anyway. So, the first I accept as my own inability to realistically predict what I can do.

    I spent two weeks in Europe, both for EuroBSDCon and meeting with other writers and publishers. If I had to fly for eight hours one way (which I detest), and shift my body clock (which I find very difficult), I was going to make the trip worthwhile. But between preparing for teaching at EuroBSDCon, physical preparations for the trip, and recovering from the trip (both physically and real life), that cost me at least a month.

    You cannot predict something like an appendolith. That’s life. I didn’t merely have an appendolith, though. I had fever and infection and all sorts of horrible ghastly things. Proper recovery took months. Plus, general anaesthesia is insidious. Even when you wake up, it muddles your brain for weeks or months afterwards.

    When life derails your goals, you get back up as soon as you can and get back on track. Maybe you can’t complete the entire goal, but you can sure do a whole bunch of it. Or maybe the deadline slips into the next year. Whatever you do, you don’t quit.

    So: I failed.

    With those things in mind, let me set some goals for 2014. I already let part of this out at NYCBSDCon, so the rest of you might as well know.

    1) I will write at least three short nonfiction books. At least one will be on OpenBSD, at least one will be on FreeBSD. At least two will see print by the end of the year.

    2) Last year’s novel will get out of my house. A couple of my author friends are encouraging me to run the novel through a publisher and have offered introductions. Their faith in my work is sincerely touching. I’m inclined to self-publish, but am keeping an open mind. We’ll see what happens. (I waited to publish this list until I finished the first draft, for those who wonder.)

    3) I’ll write at least 120,000 words of fiction. (See FAQ 9.)

    4) I will not change time zones for a conference. EuroBSDCon was great, and I’m sure that the Sofia conference will be just as grand, but that kind of travel messes me up too badly to write. I’ll be at BSDCan, but this year I’m taking the train. Because I really, really abhor flying.

    5) I’m a candidate for my dojo’s red sash test this year. If selected, I will do my best to pass. This means much practice and sweat, as the test lasts several hours. For example, my green sash test included over four hundred falls. The falling isn’t bad, but getting up again gets pretty rough. The red sash test is worse.

    My deadline for these goals in February 2015. Because my birthday is in February. Using my personal year for goals always feels better than using the calendar year.

    In a more general sense:

    I’m starting a series of short FreeBSD books, each dedicated to a single topic. Which topics will I cover? Whatever I’m working with at the moment, that’s holding still long enough for me to write about it. For example, at this moment it doesn’t make sense for me to write a book about pkgng, because pkgng is developing quickly.

    Eventually, I’ll create enough FreeBSD content to “remix” into a big FreeBSD book, probably a 3rd edition of Absolute FreeBSD.

    The small books will use the 6×9 form factor, and all be about the size of SSH Mastery. People have taken well to this size of book at the $10 ebook/$20 print price point.

    This will also let me judge which material should go into a big book. If nobody buys, say, a small FreeBSD virtualization book, it’s clear I shouldn’t put that topic into a big book, because nobody cares.

    Ideally, I’ll be able to produce a slipcase for a complete collection of small FreeBSD books. At this time, I’m planning to give them themed covers based on old pulp magazines, minus the blatant sexism and racism. (It’s been suggested by more than one person that I keep both elements but make them funny. It CAN be done, just as it is possible to make thoughtful, incisive, and honestly funny jokes about any other painful or horrifying topic. But it’s extraordinarily hard, especially for someone who looks utterly “privileged white male.” I choose to spend my energy elsewhere.) But Beastie as a hard-boiled private eye, Beastie swinging on a vine through the jungle, Beastie as the flying ace, and so on? I think that’s going to look fantastic.

    What will the OpenBSD book be? I have three ideas. I’ve caught wind of other OpenBSD books in progress, however. I need to meet with my fellow BSD authors at BSDCan 2014 and hash things out with them. It’s very important that we not step on each other’s else’s projects, especially when it’s simple enough to avoid with five minutes at the bar. That’s why I won’t do, say, a pfSense book — Chris and Jim have that territory covered quite well. I’m confident that at least one of my three ideas will be free, if for no other reason than we don’t have that many OpenBSD authors.

    I expect to let the FreeBSD Foundation have books at cost for PBS-style donation prizes. “Donate $100, and we’ll send you this $20 book!”

    I have a clever idea for using the OpenBSD book to support OpenBSD. Theo and I discussed it briefly at EuroBSDCon. I don’t know if it will actually work, mind you. But worst case, they’ll have my book in the OpenBSD bookstore, with proceeds going to OpenBSD. (For anyone who is wondering, Austin Hook is very very easy to work with. The hardest part of getting books to the OpenBSD bookstore is figuring out how to cram all the shipping information onto the CreateSpace web form, which is certainly not Austin’s fault.)

    So, is this a cynical scheme to get you to give me more money? No… and yes.

    You’ll have the option to give me any amount of money you wish, from zero up to over a hundred bucks. There’s a couple people that I suspect will buy every book, in every version. I suspect others will get a few of the small books. Others will wait for a big book. Some will buy all the small books just so they can fill a slipcase. This is about options. It’s about getting content into reader’s hands as quickly as possible.

    But if you want to give me money, I’m certainly not going to argue.

    The good news is, I now know exactly what an appendolith feels like. The next time my appendix blows up, I’ll jump on it at the earliest possible moment. Why, just today I’ve felt three twinges that might have been a faulty appendix. Catching these things early is the key to quick recovery, after all.

    Crazed Ferrets in a Berkeley Shower – 2014 edition

    With Richard Stallman’s recent raising of the flag against LLVM and Clang, I’ve heard a lot of people talking about how the “FreeBSD people are whining about the GPLv3 terrorizing them.”

    Back in 2000, I wrote an essay for Linux.com about why I like the BSD license. It’s actually stood up fairly well to the test of time, but it’s fourteen years old now. Times have changed. So have licenses.

    Normally I object to taking something with warts, dosing it with Compound-W, and sending it back out. The structure of the original still holds up pretty well, but experience and history has added new information and context. Any new essay I wrote would look an awful lot like this one.

    So, rather than try to write out my opinions from scratch, I’m taking advantage of the Internet and updating my previous essay.

    Crazed Ferrets in a Berkeley Shower (2014)

    When USL sued Berkeley over BSD UNIX, it would have been easiest and cheapest for the university to drop the BSD tapes down a deep well and forget the Computer Science Research Group ever existed. Instead, combining the academic tradition with Berkeley’s well-known liberal spirit, they fought to give their work to the world. In a decision that has caused more arguments than most in the open-source software world, they released the code under the BSD license.

    For anyone who has avoided free software licensing for the last five years, the BSD license is very simple: give the authors credit in the source code, and don’t sue the authors if it breaks. There is no requirement to distribute any modifications to anyone. Alternately, the GPL version 2 requires that source code and any modifications be made available when modified code is redistributed. GPL version 3 has additional terms addressing patent indemnification, Tivoization, and more.

    Every week sees a new argument on some public forum about how the GPL is free, and the BSD license isn’t. Someone responds that the BSD license is more free than the GPL. Eventually, someone drags out the word “communist,” someone else fires up “corporate exploitation,” and all hope for rational discussion vanishes like free Jolt cola on the trade show floor. It’s better than mud wrestling or Jerry Springer.

    Anyone familiar with my work knows that I’m on the BSD side of the fence. The GPL is not a bad thing — as an author I would argue that it’s poorly written as either a software license or a political manifesto, but the whole “share and share alike” ideal is grand. We try to teach our children the importance of sharing. I live sharing. I try to do it myself.

    The people who use the BSD license want something very different than the people who use the GPL. The license has different goals and different motivations than those of the GPL. The end goals require different licenses.

    Let’s go back in time, to the early days of BSD.

    Why would a university, especially such a famously liberal university, struggle to turn a top-notch operating system over to corporations for their pillaging pleasure? BSD UNIX gave thousands of developers the chance to innovate on a solid infrastructure. It raised the bar for minimal acceptable performance in a computer system.

    An organization can take a chunk of BSD-licensed code, shrink-wrap it, and resell it. They might make money. But if that’s all that they do, any competitor with an enhancement can take over their market and destroy them. Without real improvements and innovation, BSD-licensed code isn’t as useful as you might think.

    On the other hand, an organization can use BSD-licensed code in specific components, free programmers, and improve their products.

    If you want to build, say, a networked fax machine you can a) build an operating system from scratch, b) invest in an embedded OS such as QNX, c) use GPL-licensed code, or d) use BSD-licensed code. Most programmers would find building an operating system interesting the first time, tedious the second, and mind-numbingly dull the third. And the organization has to pay for the programmers’ time while he does it. Purchasing an operating system also raises the product’s cost. While the GPL is slowly becoming more accepted, the thought of giving away code still gives most corporate lawyers a bad case of rotating heads and the pea-soup-spews.

    By using BSD-licensed code, however, the programmer can spend his time working on the fax machine interface. He isn’t reinventing the wheel yet again. Competitors must improve, or die. The cost to the organization is reduced. The end-users get a device that is cheaper and more reliable.

    The organization must do lots of work to make this happen. Developing hardware and software for a fax machine, or any other networked device, is no easy task. All that the BSD-licensed code does is make a few select parts easy.

    Today, Intel has network appliances based on BSD. Apple has OS X. So do IBM, Netflix, and a lot of other household names. Entire businesses and careers have been built on BSD-licensed code. They’re big names now, but many of them started with some guy, a basement, and BSD. If the University of California, Berkeley, had not opened their code base under such a non-restrictive license, none of this would be possible. Such intelligent appliances would be far more expensive for both manufacturers and consumers.

    Organizations with special code, or some other kind of secret sauce, do not want to release that code to the world. If they’ve spent millions researching how to shuffle bits in a certain way, they want to get their money back. They will not use a GPL code base that requires they release their code to the world. It’s just not going to happen.

    While organizations are not required to return code to BSD-licensed projects, many do. Why?

    The biggest reasons are cheap debugging, wide test base, and ease of merging future releases.

    Open-source developers are quick to point out flaws in code, especially in a project that focuses so much attention on correctness. They will happily point out your boneheaded mistakes.

    Open-source end-users frequently run some of the hardest-working machines on the Net, and put donated code through stress tests the original authors never intended.

    Organizations don’t have to return improments to their source BSD project, but it is in their best interest — for purely commercial reasons. Hardware changes over time, as do operating systems, and this bites organizations in the future. Here’s a true story about a company’s involvement with FreeBSD. I’ve anonymized it and changed some minor details so as to not embarrass them further. (If you were there and watched the fun, please don’t name them.)

    Company X produced a product based on FreeBSD 2. It did well. They built a thriving business. Their product required changes to the FreeBSD kernel. Some of these changes were key intellectual property, while others were improvements to related systems that anyone could have done.

    They kept all their changes as a single massive patch set.

    Time passed. Hardware changed. Company X found that FreeBSD 2 wouldn’t work well for their new hardware. They needed to move to FreeBSD 4.

    All of the improvements they made needed to be ported to the new version. Their “secret sauce” had to be rewritten for the new kernel — but so did the less important changes. In many cases FreeBSD developers had improved the systems that Company X’s code patched, but in a way that was totally incompatible with their designs.

    Company X realized that if they interacted with the FreeBSD community more and sent their non-proprietary patches back to the project, future merges would be simpler. They could implement the next generation of their proprietary stuff in a way that would fit the FreeBSD development roadmap.

    They knew this… but didn’t manage to execute.

    Time passed. They had to redo their entire forward porting process for FreeBSD 6.

    I don’t know how many times these poor bastards went through this, but I’m glad to say they’re now consistent and regular FreeBSD contributors. (Pain is a wonderful teacher, but nobody wants to go to his school.)

    For me, knowing that Company X “gets it” and donates code back to FreeBSD makes me look upon them much more kindly. I know other organizations have taken BSD code and not returned anything. They’re not required to, but they don’t stick in my mind as one of the good guys. I also chuckle when I imagine them merging their code forward.

    Some people fear that the BSD license allows companies to take their work and make it proprietary. Source code isn’t a limited resource; any number of people can have it, use it, or improve it. The two programs that hold the Internet together, BIND and Sendmail, had BSD licenses for many years. If a company could seize control of one or another of these programs, the Internet would be their oyster and we would be their slaves. People have tried. People have spent lots of money trying to execute this nefarious master plan. It hasn’t happened. It hasn’t happened because it can’t.

    Even if a Big Evil Software Company (tm., pat. pend.) wanted to assimilate BIND, they would have to convince the current BIND users to pay for something they could still get for free. Any vital, but proprietary, improvement to BIND will be duplicated quickly by open-source developers. A closed-source development process cannot withstand the onslaught of open source when it comes to vital core functions, especially when you start with the same code base. Those of us watching the race would have to content ourselves with pointing, laughing, and buying the volunteers beer.

    Look at TCP/IP. It was originally implemented in BSD. The BSD implementation became the standard. Everybody uses it. If you write your own stack, it’s expected to be compatible with BSD TCP/IP. Or everyone calls you a loser and says your IP stack sucks. Which it does.

    Closed source companies can and do compete on the bells-and-whistles front. And people are willing to pay for the pretty point-and-click interfaces. Heck, more than once I’ve snarled obscenities while trying to make Sendmail relay mail for a specific type of client, and if you asked me at the right moment I would have plopped down a credit card to make the problem go away.

    Now, users without source code will have a hard time fixing bugs or adding features themselves. Most organizations today have trouble finding people who can make reports off their Access databases, let alone people who could fix device drivers. Organizations who have that talent use it. Most organizations fix problems by asking the vendor.

    And no code, whether GPLd or BSD’d, can make up for an organization that does not respond to its customers. Starting with known working code can give them a head start, but a non-responsive organization still won’t go far. You probably gripe about Microsoft and Apple, but the average user wants an operating system where they can easily install a kitten screen saver with extra spyware. The Windows interface might not work well, but it is easy. The average corporate customers want simple more than they want performance.

    From the developer’s viewpoint, code under the BSD license provides a certain “minimal acceptable level” of software quality. Can you write an IP stack better than the one in OpenBSD? If not, why bother? Isn’t there something you’d rather do than handle exceptions for SYNs, ACKs, and RSTs, then return every six months to deal with the latest denial-of-service attack?

    Hundreds of organizations have made this same decision; the BSD IP stack has found its way into countless products. Almost any network device that isn’t labeled Linux or System V uses BSD. Some products even combine them; the RTEMS real-time operating system, which is distributed under the GPL, incorporated the BSD IP stack.

    Things could have been very different.

    Imagine someone travels back in time to the point where the Berkeley regents were choosing the license for their UNIX code. Our time traveler is a very persuasive man, and convinces several regents that they should use the GPL. Most of the others receive a cup of coffee spiked with Richard Stallman’s messenger RNA. The last holdout is distracted on the morning of the final vote by the half dozen crazed ferrets that somehow got in his shower. BSD UNIX receives the GPL.

    Consider how the GPL would have been received in the corporate boardroom of the 1980s. BSD UNIX never would have been used.

    Organizations frequently build software features such that they work. That’s very different from “they work correctly,” but that’s what we would have gotten.

    Imagine SunOS shipping without BSD’s vi; you’d have ed as the default editor. Imagine Digital UNIX with a “good enough” virtual memory system. SVR4 wouldn’t have had long filenames, or job control. The standard Unix File System is a BSD invention; SVR4 would have shipped with the S51K filesystem. A power failure would have meant hours of work using tools like fsdb and clri. If you’ve never heard of fsdb and clri, you don’t want to know about them. Trust me on this one.

    If all that didn’t make you flinch: imagine the toll in sheer human suffering if Microsoft hadn’t used the BSD TCP/IP stack.

    If Windows was the only competition faced by Linux, Linux would not have come so far. Today, the Linux and BSD camps work every day to outpace each other. We don’t know where that race will end. We do know that the software at the end of it will work.

    Today, the attitude towards open source software is quite different than it was in the 1980s. The corporate world accepts open source, and even the GPL. Many will not release their code under the GPL, but they contribute many improvements back to to the BSD world and from there to everyone.

    We can never be sure what the world will think tomorrow, or next year. After all, forty years ago everyone knew you needed source code to do anything useful. Twenty years ago, source code was irrelevant for many organizations. We’ve returned to the open source world, but the wheel might just keep turning through open source and back the other way.

    Think it can’t happen? Lots of people would like to make Vernor Vinge’s Ubiquitous Law Enforcement real. Many of them have a lot of money. There are alternative scenarios that would crush or marginalize open source operating systems, much as car tinkerers are a fringe group today.

    In 2020, or 2030, we might hear the words “Anyone remember the open source fad?” I don’t think it will happen, but history is littered with the rotting corpses of invulnerable juggernauts.

    Even if the open source movement collapses, BSD-licensed code will still be used. The open-source BSD groups will have support from people smart enough to recognize the benefits of open source, but who are hamstrung by organizational policy or legal problems.

    Average, everyday people, whose only interest in a computer is from 9 to 5, Monday through Friday, benefit. Imagine if your code could, every day, save one thousand people five minutes of annoyance, frustration, or downright hysteria, without them having to know or do anything. BSD-licensed code saves more than that, at no further cost to anyone.

    The BSD license is not about open source. Open source is simply a tool to get better software. It’s not about ideology. It’s about correct, stable, and above all, better software, not just for computer-literate individuals but for everyone. It’s about giving users peace of mind, and letting developers do original work.

    BSD is about making the world a better place. For everyone.

    updated 2014-01-31: corrected some GPLv3 terms

    Why I don’t have ads here

    Another “put this up so I can point to it later” post.

    Now and then someone tells me that I should put ads on my blog. Some of the articles get a ridiculous number of search engine hits, and I could probably add another (small) income stream there.

    I’m not morally opposed to the very idea of advertising-supported web sites or, indeed, advertising in general. I’m opposed to leaking information about my readers, however. And that’s what the advertisers actually pay for.

    I don’t oppose sharing information, mind you. If I could easily customize this blog so you only saw what you wanted without hooking you into the government/corporate exploitation network, I would. (It can be done, mind you, but it’s not trivial, so it’s not worth my time.)

    That’s why I let people send me money over at my online bookstore. I’d rather perform on a streetcorner with my hat in front of me than feed human flow data to anonymous information leeches.

    Of course, that raises the question of “why don’t you have a tip jar link on every blog post?” I write this blog to remind me what I did and to promote my work. If you can make use of either, fine — but that’s incidental to my purposes. Rather like OpenBSD, actually.

    Jan 2014 Java update broke me

    So I’m trying to upgrade my Ansible server to the newest OpenBSD snapshot, which involves working at the console. I go to my virtual server control panel, click on the link to the Java applet, and get told that Java won’t run this application.

    Turns out that Java has trusted self-signed certificates for applications until now, relying on blacklists rather than whitelists. I simultaneously applaud this move away from enumerating badness and condemn them for temporarily inconveniencing me.

    To whitelist a specific site, open the Java configuration applet. For Windows users, this is the Java Control Panel. Open the Security tab. About 2/3 of the way down, there’s an “Edit site list” option. Add the desired web site.

    Java will then run applets from that web site.