Installing FreeBSD 10 to ZFS with a script

Well, partially scripted, that is.

For installing large numbers of identical machines, proceed directly to the PC-BSD installer. It’s easy to configure, very reliable, and generally just rocks. If you’re accustomed to automatic installers like Kickstart, you’ll find the PC-BSD installer trivially easy.

I frequently have to install non-identical machines for special purposes, such as testing or unique file stores or EDI. Most of these are virtual machines. It seems that ZFS filesystems compress really really well, simplifying backing up the VMs.

And there, the FreeBSD 10 installer’s ZFS features don’t quite cut it. The installer lets you create a single large ZFS without leaving the GUI. I want a more complicated ZFS setup, based on the FreeBSD Root on ZFS wiki page. This process involves a whole lot of typing. I normally install servers when I’m too brain dead to do any real work, so I need to minimize the opportunity for errors.

Fortunately, you can script all the disk and ZFS setup. Here’s my script. If it looks familiar, well, it should: it’s ripped raw and bleeding from the wiki instructions and wrapped up with /bin/sh -x. (I use -x because I want to see how the script runs.)

#!/bin/sh -x

#Auto-divides a ZFS install.
#ZFS permissions stolen from
#https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/9.0-RELEASE

#edit:
#disk device name
#parameters for your zpool type
#your pool name
#swap space

#we're installing
sysctl kern.geom.debugflags=0x10

gpart destroy -F vtbd0
gpart create -s gpt vtbd0
gpart add -s 222 -a 4k -t freebsd-boot -l boot0 vtbd0

gpart add -s 1g -a 4k -t freebsd-swap -l swap0 vtbd0
gpart add -a 4k -t freebsd-zfs -l disk0 vtbd0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 vtbd0
gnop create -S 4096 /dev/gpt/disk0

kldload zfs
zpool create -f -o altroot=/mnt -O canmount=off -m none zroot /dev/gpt/disk0.nop

zfs set checksum=fletcher4 zroot
zfs set atime=off zroot

zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default
zfs create -o mountpoint=/tmp -o compression=lzjb -o setuid=off zroot/tmp
chmod 1777 /mnt/tmp

zfs create -o mountpoint=/usr zroot/usr
zfs create zroot/usr/local

zfs create -o mountpoint=/home -o setuid=off zroot/home
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages

zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src
zfs create zroot/usr/obj

zfs create -o mountpoint=/var zroot/var
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
chmod 1777 /mnt/var/tmp
zpool set bootfs=zroot/ROOT/default zroot

cat << EOF > /tmp/bsdinstall_etc/fstab
/dev/gpt/swap0 none swap sw 0 0
EOF

exit

How do I use this?

When the installer asks me how I want to partition the disk, I exit to the shell, configure the network, and get the script onto the target system.

# dhclient vtnet0
# cd /tmp
# fetch http://www-old.michaelwlucas.com/zfsinstall.sh
# chmod 755 zfsinstall.sh
# ./zfsinstall.sh

Sit back and watch it run. When the script finishes, exit from the shell and let the installer unpack the files. When you’re offered a shell to perform post-install configuration, take it and run the post-install ZFS setup commands.

# mount -t devfs devfs /dev
# echo 'zfs_enable="YES"' >> /etc/rc.conf
# echo 'zfs_load="YES"' >> /boot/loader.conf
# zfs set readonly=on zroot/var/empty

Reboot into your fine-grained ZFS filesystem installation.

To use this script yourself, you’ll need to check the disk device name and the type of zpool you want to create. But this will hopefully get you started.

I would really like to see the default FreeBSD installer create finer grained ZFS filesystems. I’m told that day is coming.

ifup-local on bridge members on CentOS

I run a bunch of CentOS 6 physical servers as QEMU virtualization devices. These hosts have two NICs, one for management and one for virtual machine bridges.

When you use Linux for virtualization, it’s important to increase the amount of memory for network transmit and receive buffers. You also need to disable GSO and TSO, to improve performance and to avoid gigabytes of kernel error messages every day. You can do this with ethtool(8). First, let’s check the existing ring sizes.

# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX: 16384
RX Mini: 0
RX Jumbo: 0
TX: 16384
Current hardware settings:
RX: 512
RX Mini: 0
RX Jumbo: 0
TX: 512

Similarly, use ethtool -k eth0 to check GSO and TSO settings.

The card is using much less memory than it can. When you have a bunch of virtual machines pouring data through the card, you want the card to work as efficiently as possible. Fixing this on a running system is easy enough:

# ethtool -G eth0 tx 16384 rx 16384
# ethtool -K eth0 gso off tso off

Repeat the process for eth1.

How do you make this happen automatically at boot? Adding the commands to /etc/rc.local isn’t reliable. By the time the system gets that much stuff running, the ethtool command might fail with a “Cannot allocate memory” error. If you try again it’ll probably work, but it’s not deterministic. And I’m against running a single command four times in rc.local in the hopes that one of them will work.

Enter /sbin/ifup-local. CentOS runs this script after bringing up an interface, with the interface name as an argument. The problem is, it doesn’t run this script on bridge member interfaces. We can adjust eth0 and br0 at boot just fine, but eth1 (the physical interface underlying br0) doesn’t get run.

You can’t run ethtool -G eth0 tx 16384 rx 16384 on br0. Interface br0 doesn’t have any transmit or receive rings. It’s a logical interface. You can disable TSO and GSO on br0, but that won’t disable it on eth1. You can’t wait to reconfigure eth1 in rc.local until the system is running, because increasing the memory doesn’t always work once the system is running full-out multiuser. And Red Hat says this is by design. Apparently network bridges on CentOS/Red Hat are supposed to perform poorly. That’s good to know.

So, what to do?

I adjust the eth1 ring size in ifup-local when bringing up br0, but before any processes send any traffic over the bridge. My /sbin/ifup-local looks like this:

#!/bin/bash

case "$1" in
eth0)
echo "Configuring eth0..."
/sbin/ethtool -G eth0 tx 16384 rx 16384
/sbin/ethtool -K eth0 gso off tso off
;;

br0)
echo "Configuring br0..."
/sbin/ethtool -G eth1 tx 16384 rx 16384
/sbin/ethtool -K eth1 gso off tso off
/sbin/ethtool -K br0 gso off tso off
;;

esac
exit 0

This appears to work consistently. Of course, the values for the NIC need to be set on a per-machine basis. I have Ansible do that work for me.

Hopefully, this will save someone else the pain I’ve been through trying to make this work…

New reviews

There’s been a few new reviews out lately. First, two from Grant Taylor, on Sudo Mastery and SSH Mastery. Thank you, Grant!

Yesterday, a review of Sudo Mastery appeared on Slashdot. I haven’t been reviewed on Slashdot since Absolute OpenBSD came out. No, not the second edition–the original, in 2003. So this is cool. Thank you, “Saint Aardvark.” (Yes, I can figure out his real name, but if he goes by that, who am I to argue?)

As a result of these reviews, I now simultaneously have the #1 and #4 best-seller slots in Amazon’s Unix category.

I really want to thank everyone who takes the time to review my books — or, indeed, any books. Reviews drive sales. Sales mean that authors can afford to write books instead of washing dishes at the Burger Hut (which is all that most of us are qualified for in the real world). If you enjoy a book, and want to thank the author, take a moment to do so publicly.

And now back to writing more books…

2 titles in Amazon's top 10

2013 Failures and 2014 Goals

I set goals for 2013. And I failed to meet them. I promised three short nonfiction books, Absolute OpenBSD 2nd edition, and a novel. You got AO2e and two short nonfiction books, DNSSEC Mastery and Sudo Mastery.

While setting goals is important, exploring why you fail to meet those goals is just as important. Driving factors behind these goals boil down to three things.

  • These were pretty ambitious goals
  • Traveled to EuroBSDCon in September
  • January’s emergency appendectomy
  • I knew this was ambitious beforehand, but decided to try for it anyway. So, the first I accept as my own inability to realistically predict what I can do.

    I spent two weeks in Europe, both for EuroBSDCon and meeting with other writers and publishers. If I had to fly for eight hours one way (which I detest), and shift my body clock (which I find very difficult), I was going to make the trip worthwhile. But between preparing for teaching at EuroBSDCon, physical preparations for the trip, and recovering from the trip (both physically and real life), that cost me at least a month.

    You cannot predict something like an appendolith. That’s life. I didn’t merely have an appendolith, though. I had fever and infection and all sorts of horrible ghastly things. Proper recovery took months. Plus, general anaesthesia is insidious. Even when you wake up, it muddles your brain for weeks or months afterwards.

    When life derails your goals, you get back up as soon as you can and get back on track. Maybe you can’t complete the entire goal, but you can sure do a whole bunch of it. Or maybe the deadline slips into the next year. Whatever you do, you don’t quit.

    So: I failed.

    With those things in mind, let me set some goals for 2014. I already let part of this out at NYCBSDCon, so the rest of you might as well know.

    1) I will write at least three short nonfiction books. At least one will be on OpenBSD, at least one will be on FreeBSD. At least two will see print by the end of the year.

    2) Last year’s novel will get out of my house. A couple of my author friends are encouraging me to run the novel through a publisher and have offered introductions. Their faith in my work is sincerely touching. I’m inclined to self-publish, but am keeping an open mind. We’ll see what happens. (I waited to publish this list until I finished the first draft, for those who wonder.)

    3) I’ll write at least 120,000 words of fiction. (See FAQ 9.)

    4) I will not change time zones for a conference. EuroBSDCon was great, and I’m sure that the Sofia conference will be just as grand, but that kind of travel messes me up too badly to write. I’ll be at BSDCan, but this year I’m taking the train. Because I really, really abhor flying.

    5) I’m a candidate for my dojo’s red sash test this year. If selected, I will do my best to pass. This means much practice and sweat, as the test lasts several hours. For example, my green sash test included over four hundred falls. The falling isn’t bad, but getting up again gets pretty rough. The red sash test is worse.

    My deadline for these goals in February 2015. Because my birthday is in February. Using my personal year for goals always feels better than using the calendar year.

    In a more general sense:

    I’m starting a series of short FreeBSD books, each dedicated to a single topic. Which topics will I cover? Whatever I’m working with at the moment, that’s holding still long enough for me to write about it. For example, at this moment it doesn’t make sense for me to write a book about pkgng, because pkgng is developing quickly.

    Eventually, I’ll create enough FreeBSD content to “remix” into a big FreeBSD book, probably a 3rd edition of Absolute FreeBSD.

    The small books will use the 6×9 form factor, and all be about the size of SSH Mastery. People have taken well to this size of book at the $10 ebook/$20 print price point.

    This will also let me judge which material should go into a big book. If nobody buys, say, a small FreeBSD virtualization book, it’s clear I shouldn’t put that topic into a big book, because nobody cares.

    Ideally, I’ll be able to produce a slipcase for a complete collection of small FreeBSD books. At this time, I’m planning to give them themed covers based on old pulp magazines, minus the blatant sexism and racism. (It’s been suggested by more than one person that I keep both elements but make them funny. It CAN be done, just as it is possible to make thoughtful, incisive, and honestly funny jokes about any other painful or horrifying topic. But it’s extraordinarily hard, especially for someone who looks utterly “privileged white male.” I choose to spend my energy elsewhere.) But Beastie as a hard-boiled private eye, Beastie swinging on a vine through the jungle, Beastie as the flying ace, and so on? I think that’s going to look fantastic.

    What will the OpenBSD book be? I have three ideas. I’ve caught wind of other OpenBSD books in progress, however. I need to meet with my fellow BSD authors at BSDCan 2014 and hash things out with them. It’s very important that we not step on each other’s else’s projects, especially when it’s simple enough to avoid with five minutes at the bar. That’s why I won’t do, say, a pfSense book — Chris and Jim have that territory covered quite well. I’m confident that at least one of my three ideas will be free, if for no other reason than we don’t have that many OpenBSD authors.

    I expect to let the FreeBSD Foundation have books at cost for PBS-style donation prizes. “Donate $100, and we’ll send you this $20 book!”

    I have a clever idea for using the OpenBSD book to support OpenBSD. Theo and I discussed it briefly at EuroBSDCon. I don’t know if it will actually work, mind you. But worst case, they’ll have my book in the OpenBSD bookstore, with proceeds going to OpenBSD. (For anyone who is wondering, Austin Hook is very very easy to work with. The hardest part of getting books to the OpenBSD bookstore is figuring out how to cram all the shipping information onto the CreateSpace web form, which is certainly not Austin’s fault.)

    So, is this a cynical scheme to get you to give me more money? No… and yes.

    You’ll have the option to give me any amount of money you wish, from zero up to over a hundred bucks. There’s a couple people that I suspect will buy every book, in every version. I suspect others will get a few of the small books. Others will wait for a big book. Some will buy all the small books just so they can fill a slipcase. This is about options. It’s about getting content into reader’s hands as quickly as possible.

    But if you want to give me money, I’m certainly not going to argue.

    The good news is, I now know exactly what an appendolith feels like. The next time my appendix blows up, I’ll jump on it at the earliest possible moment. Why, just today I’ve felt three twinges that might have been a faulty appendix. Catching these things early is the key to quick recovery, after all.

    Crazed Ferrets in a Berkeley Shower – 2014 edition

    With Richard Stallman’s recent raising of the flag against LLVM and Clang, I’ve heard a lot of people talking about how the “FreeBSD people are whining about the GPLv3 terrorizing them.”

    Back in 2000, I wrote an essay for Linux.com about why I like the BSD license. It’s actually stood up fairly well to the test of time, but it’s fourteen years old now. Times have changed. So have licenses.

    Normally I object to taking something with warts, dosing it with Compound-W, and sending it back out. The structure of the original still holds up pretty well, but experience and history has added new information and context. Any new essay I wrote would look an awful lot like this one.

    So, rather than try to write out my opinions from scratch, I’m taking advantage of the Internet and updating my previous essay.

    Crazed Ferrets in a Berkeley Shower (2014)

    When USL sued Berkeley over BSD UNIX, it would have been easiest and cheapest for the university to drop the BSD tapes down a deep well and forget the Computer Science Research Group ever existed. Instead, combining the academic tradition with Berkeley’s well-known liberal spirit, they fought to give their work to the world. In a decision that has caused more arguments than most in the open-source software world, they released the code under the BSD license.

    For anyone who has avoided free software licensing for the last five years, the BSD license is very simple: give the authors credit in the source code, and don’t sue the authors if it breaks. There is no requirement to distribute any modifications to anyone. Alternately, the GPL version 2 requires that source code and any modifications be made available when modified code is redistributed. GPL version 3 has additional terms addressing patent indemnification, Tivoization, and more.

    Every week sees a new argument on some public forum about how the GPL is free, and the BSD license isn’t. Someone responds that the BSD license is more free than the GPL. Eventually, someone drags out the word “communist,” someone else fires up “corporate exploitation,” and all hope for rational discussion vanishes like free Jolt cola on the trade show floor. It’s better than mud wrestling or Jerry Springer.

    Anyone familiar with my work knows that I’m on the BSD side of the fence. The GPL is not a bad thing — as an author I would argue that it’s poorly written as either a software license or a political manifesto, but the whole “share and share alike” ideal is grand. We try to teach our children the importance of sharing. I live sharing. I try to do it myself.

    The people who use the BSD license want something very different than the people who use the GPL. The license has different goals and different motivations than those of the GPL. The end goals require different licenses.

    Let’s go back in time, to the early days of BSD.

    Why would a university, especially such a famously liberal university, struggle to turn a top-notch operating system over to corporations for their pillaging pleasure? BSD UNIX gave thousands of developers the chance to innovate on a solid infrastructure. It raised the bar for minimal acceptable performance in a computer system.

    An organization can take a chunk of BSD-licensed code, shrink-wrap it, and resell it. They might make money. But if that’s all that they do, any competitor with an enhancement can take over their market and destroy them. Without real improvements and innovation, BSD-licensed code isn’t as useful as you might think.

    On the other hand, an organization can use BSD-licensed code in specific components, free programmers, and improve their products.

    If you want to build, say, a networked fax machine you can a) build an operating system from scratch, b) invest in an embedded OS such as QNX, c) use GPL-licensed code, or d) use BSD-licensed code. Most programmers would find building an operating system interesting the first time, tedious the second, and mind-numbingly dull the third. And the organization has to pay for the programmers’ time while he does it. Purchasing an operating system also raises the product’s cost. While the GPL is slowly becoming more accepted, the thought of giving away code still gives most corporate lawyers a bad case of rotating heads and the pea-soup-spews.

    By using BSD-licensed code, however, the programmer can spend his time working on the fax machine interface. He isn’t reinventing the wheel yet again. Competitors must improve, or die. The cost to the organization is reduced. The end-users get a device that is cheaper and more reliable.

    The organization must do lots of work to make this happen. Developing hardware and software for a fax machine, or any other networked device, is no easy task. All that the BSD-licensed code does is make a few select parts easy.

    Today, Intel has network appliances based on BSD. Apple has OS X. So do IBM, Netflix, and a lot of other household names. Entire businesses and careers have been built on BSD-licensed code. They’re big names now, but many of them started with some guy, a basement, and BSD. If the University of California, Berkeley, had not opened their code base under such a non-restrictive license, none of this would be possible. Such intelligent appliances would be far more expensive for both manufacturers and consumers.

    Organizations with special code, or some other kind of secret sauce, do not want to release that code to the world. If they’ve spent millions researching how to shuffle bits in a certain way, they want to get their money back. They will not use a GPL code base that requires they release their code to the world. It’s just not going to happen.

    While organizations are not required to return code to BSD-licensed projects, many do. Why?

    The biggest reasons are cheap debugging, wide test base, and ease of merging future releases.

    Open-source developers are quick to point out flaws in code, especially in a project that focuses so much attention on correctness. They will happily point out your boneheaded mistakes.

    Open-source end-users frequently run some of the hardest-working machines on the Net, and put donated code through stress tests the original authors never intended.

    Organizations don’t have to return improments to their source BSD project, but it is in their best interest — for purely commercial reasons. Hardware changes over time, as do operating systems, and this bites organizations in the future. Here’s a true story about a company’s involvement with FreeBSD. I’ve anonymized it and changed some minor details so as to not embarrass them further. (If you were there and watched the fun, please don’t name them.)

    Company X produced a product based on FreeBSD 2. It did well. They built a thriving business. Their product required changes to the FreeBSD kernel. Some of these changes were key intellectual property, while others were improvements to related systems that anyone could have done.

    They kept all their changes as a single massive patch set.

    Time passed. Hardware changed. Company X found that FreeBSD 2 wouldn’t work well for their new hardware. They needed to move to FreeBSD 4.

    All of the improvements they made needed to be ported to the new version. Their “secret sauce” had to be rewritten for the new kernel — but so did the less important changes. In many cases FreeBSD developers had improved the systems that Company X’s code patched, but in a way that was totally incompatible with their designs.

    Company X realized that if they interacted with the FreeBSD community more and sent their non-proprietary patches back to the project, future merges would be simpler. They could implement the next generation of their proprietary stuff in a way that would fit the FreeBSD development roadmap.

    They knew this… but didn’t manage to execute.

    Time passed. They had to redo their entire forward porting process for FreeBSD 6.

    I don’t know how many times these poor bastards went through this, but I’m glad to say they’re now consistent and regular FreeBSD contributors. (Pain is a wonderful teacher, but nobody wants to go to his school.)

    For me, knowing that Company X “gets it” and donates code back to FreeBSD makes me look upon them much more kindly. I know other organizations have taken BSD code and not returned anything. They’re not required to, but they don’t stick in my mind as one of the good guys. I also chuckle when I imagine them merging their code forward.

    Some people fear that the BSD license allows companies to take their work and make it proprietary. Source code isn’t a limited resource; any number of people can have it, use it, or improve it. The two programs that hold the Internet together, BIND and Sendmail, had BSD licenses for many years. If a company could seize control of one or another of these programs, the Internet would be their oyster and we would be their slaves. People have tried. People have spent lots of money trying to execute this nefarious master plan. It hasn’t happened. It hasn’t happened because it can’t.

    Even if a Big Evil Software Company (tm., pat. pend.) wanted to assimilate BIND, they would have to convince the current BIND users to pay for something they could still get for free. Any vital, but proprietary, improvement to BIND will be duplicated quickly by open-source developers. A closed-source development process cannot withstand the onslaught of open source when it comes to vital core functions, especially when you start with the same code base. Those of us watching the race would have to content ourselves with pointing, laughing, and buying the volunteers beer.

    Look at TCP/IP. It was originally implemented in BSD. The BSD implementation became the standard. Everybody uses it. If you write your own stack, it’s expected to be compatible with BSD TCP/IP. Or everyone calls you a loser and says your IP stack sucks. Which it does.

    Closed source companies can and do compete on the bells-and-whistles front. And people are willing to pay for the pretty point-and-click interfaces. Heck, more than once I’ve snarled obscenities while trying to make Sendmail relay mail for a specific type of client, and if you asked me at the right moment I would have plopped down a credit card to make the problem go away.

    Now, users without source code will have a hard time fixing bugs or adding features themselves. Most organizations today have trouble finding people who can make reports off their Access databases, let alone people who could fix device drivers. Organizations who have that talent use it. Most organizations fix problems by asking the vendor.

    And no code, whether GPLd or BSD’d, can make up for an organization that does not respond to its customers. Starting with known working code can give them a head start, but a non-responsive organization still won’t go far. You probably gripe about Microsoft and Apple, but the average user wants an operating system where they can easily install a kitten screen saver with extra spyware. The Windows interface might not work well, but it is easy. The average corporate customers want simple more than they want performance.

    From the developer’s viewpoint, code under the BSD license provides a certain “minimal acceptable level” of software quality. Can you write an IP stack better than the one in OpenBSD? If not, why bother? Isn’t there something you’d rather do than handle exceptions for SYNs, ACKs, and RSTs, then return every six months to deal with the latest denial-of-service attack?

    Hundreds of organizations have made this same decision; the BSD IP stack has found its way into countless products. Almost any network device that isn’t labeled Linux or System V uses BSD. Some products even combine them; the RTEMS real-time operating system, which is distributed under the GPL, incorporated the BSD IP stack.

    Things could have been very different.

    Imagine someone travels back in time to the point where the Berkeley regents were choosing the license for their UNIX code. Our time traveler is a very persuasive man, and convinces several regents that they should use the GPL. Most of the others receive a cup of coffee spiked with Richard Stallman’s messenger RNA. The last holdout is distracted on the morning of the final vote by the half dozen crazed ferrets that somehow got in his shower. BSD UNIX receives the GPL.

    Consider how the GPL would have been received in the corporate boardroom of the 1980s. BSD UNIX never would have been used.

    Organizations frequently build software features such that they work. That’s very different from “they work correctly,” but that’s what we would have gotten.

    Imagine SunOS shipping without BSD’s vi; you’d have ed as the default editor. Imagine Digital UNIX with a “good enough” virtual memory system. SVR4 wouldn’t have had long filenames, or job control. The standard Unix File System is a BSD invention; SVR4 would have shipped with the S51K filesystem. A power failure would have meant hours of work using tools like fsdb and clri. If you’ve never heard of fsdb and clri, you don’t want to know about them. Trust me on this one.

    If all that didn’t make you flinch: imagine the toll in sheer human suffering if Microsoft hadn’t used the BSD TCP/IP stack.

    If Windows was the only competition faced by Linux, Linux would not have come so far. Today, the Linux and BSD camps work every day to outpace each other. We don’t know where that race will end. We do know that the software at the end of it will work.

    Today, the attitude towards open source software is quite different than it was in the 1980s. The corporate world accepts open source, and even the GPL. Many will not release their code under the GPL, but they contribute many improvements back to to the BSD world and from there to everyone.

    We can never be sure what the world will think tomorrow, or next year. After all, forty years ago everyone knew you needed source code to do anything useful. Twenty years ago, source code was irrelevant for many organizations. We’ve returned to the open source world, but the wheel might just keep turning through open source and back the other way.

    Think it can’t happen? Lots of people would like to make Vernor Vinge’s Ubiquitous Law Enforcement real. Many of them have a lot of money. There are alternative scenarios that would crush or marginalize open source operating systems, much as car tinkerers are a fringe group today.

    In 2020, or 2030, we might hear the words “Anyone remember the open source fad?” I don’t think it will happen, but history is littered with the rotting corpses of invulnerable juggernauts.

    Even if the open source movement collapses, BSD-licensed code will still be used. The open-source BSD groups will have support from people smart enough to recognize the benefits of open source, but who are hamstrung by organizational policy or legal problems.

    Average, everyday people, whose only interest in a computer is from 9 to 5, Monday through Friday, benefit. Imagine if your code could, every day, save one thousand people five minutes of annoyance, frustration, or downright hysteria, without them having to know or do anything. BSD-licensed code saves more than that, at no further cost to anyone.

    The BSD license is not about open source. Open source is simply a tool to get better software. It’s not about ideology. It’s about correct, stable, and above all, better software, not just for computer-literate individuals but for everyone. It’s about giving users peace of mind, and letting developers do original work.

    BSD is about making the world a better place. For everyone.

    updated 2014-01-31: corrected some GPLv3 terms

    Why I don’t have ads here

    Another “put this up so I can point to it later” post.

    Now and then someone tells me that I should put ads on my blog. Some of the articles get a ridiculous number of search engine hits, and I could probably add another (small) income stream there.

    I’m not morally opposed to the very idea of advertising-supported web sites or, indeed, advertising in general. I’m opposed to leaking information about my readers, however. And that’s what the advertisers actually pay for.

    I don’t oppose sharing information, mind you. If I could easily customize this blog so you only saw what you wanted without hooking you into the government/corporate exploitation network, I would. (It can be done, mind you, but it’s not trivial, so it’s not worth my time.)

    That’s why I let people send me money over at my online bookstore. I’d rather perform on a streetcorner with my hat in front of me than feed human flow data to anonymous information leeches.

    Of course, that raises the question of “why don’t you have a tip jar link on every blog post?” I write this blog to remind me what I did and to promote my work. If you can make use of either, fine — but that’s incidental to my purposes. Rather like OpenBSD, actually.

    Jan 2014 Java update broke me

    So I’m trying to upgrade my Ansible server to the newest OpenBSD snapshot, which involves working at the console. I go to my virtual server control panel, click on the link to the Java applet, and get told that Java won’t run this application.

    Turns out that Java has trusted self-signed certificates for applications until now, relying on blacklists rather than whitelists. I simultaneously applaud this move away from enumerating badness and condemn them for temporarily inconveniencing me.

    To whitelist a specific site, open the Java configuration applet. For Windows users, this is the Java Control Panel. Open the Security tab. About 2/3 of the way down, there’s an “Edit site list” option. Add the desired web site.

    Java will then run applets from that web site.

    Ansible and PF, plus NTP

    It seems that ntpd has turned into the latest DDOS amplifier. I run a lot of servers, and most of them use the standard ntp client. I need to verify that none of my servers can be used for DDOS amplification. To do this, I need to give all the clients a standard NTP configuration, pointing at my personal NTP servers.

    While my internal addresses need access to the port 123 on my servers, the public doesn’t. And I occasionally add internal addresses. Automating PF and NTP configuration via Ansible will simplify my life in the future.

    I’ve used Ansible templates to configure services before, but packet filters are a little different. Packet filtering rules involve lots of information about the local host, such as interface names and the various system roles. It’s entirely possible to write an Ansible template that expresses your PF ruleset, it just took a little work.

    First, I define an Ansible group for the NTP servers (as well as other server duties). The time servers run FreeBSD 9.2.

    Here’s the playbook, with added NTP.

    ---
    - hosts: ntp-servers
      user: ansible
      sudo: yes
    
      tasks:
    
      - name: enable ntpdate
        action: command sysrc ntpdate_enable=yes
      - name: enable ntpdate server
        action: command sysrc ntpdate_hosts=pool.ntp.org
      - name: enable ntpd
        action: command sysrc ntpd_enable=yes
    
      - name: copy ntp.server.conf to servers
        action: copy src=/home/ansible/freebsd/etc/ntp.server.conf
          dest=/etc/ntp.conf owner=root group=wheel mode=0644
        notify:
          - restart ntpd
    
      - include: tasks/pf-compile.yml
    
      handlers:
    
        - include: handlers/restarts.yml

    Simple enough, no?

    Except there’s nothing in here about the packet filter. Or restarting ntp.

    These functions are pretty common, so I’ve moved them to separate files. I might need to rebuild the packet filter rules for any number of playbooks, after all.

    The file tasks/pf-compile.yml looks like this.

    ---
    #build pf.conf from template
    
      - name: configure firewall
        template: src=/home/ansible/freebsd/etc/pf.conf.j2
          dest=/etc/pf.conf owner=0 group=0 mode=0444
          validate='/sbin/pfctl -nf %s'
        notify:
          - reload pf

    This task uses a jinja2 template to build a pf.conf specifically for this host, copies it to the host, validates its syntax, puts it in place, and triggers a PF reload. Always validate your files before deploying them. Ansible doesn’t prevent mistakes, but rather allows you to deploy mistakes faster than ever.

    Similarly, I’ve split the “restart services” handlers off into the file handlers/restarts.yml. Here are the relevant bits.

    ---
    #restart assorted services
    
      - name: restart ntpd
        service: name=ntpd state=restarted
    
      - name: reload pf
        action: shell /sbin/pfctl -f /etc/pf.conf

    So, where is this firewall template? That’s probably what dragged you here.

    #{{ ansible_managed }}
    #$Id: pf.conf.j2,v 1.2 2014/01/16 16:10:54 mwlucas Exp $
    
    ext_if="{{ ansible_default_ipv4.device }}"
    
    include "/etc/pf.mgmt.conf"
    include "/etc/pf.ournets.conf"
    
    set block-policy return
    set loginterface $ext_if
    set skip on lo0
    
    scrub in all
    block in all
    
    pass in on $ext_if proto icmp all
    pass in on $ext_if proto icmp6 all
    
    #this host may initiate anything
    pass out on $ext_if from any to any
    
    #mgmt networks can talk to this host on any service
    pass in on $ext_if from  to any
    
    #Allowed services, in port order
    
    {% if inventory_hostname in groups['dns'] %}
    #DNS access
    pass in on $ext_if proto {tcp, udp} from any to any port 53
    {% endif %}
    
    {% if inventory_hostname in groups['tftpd'] %}
    #allow tftp from the world
    pass in on $ext_if proto udp from any to any port 69
    {% endif %}
    
    {% if inventory_hostname in groups['ntp-servers'] %}
    #allow time from our networks
    pass in on $ext_if proto udp from  to any port 123
    {% endif %}
    
    #end of services

    The first bit of new (to me) trickery in this is getting the interface name. I use an Ansible-provided variable for this. Get the complete list of Ansible-provided variables for a host by running ansible -m setup hostname. The variable ansible_default_ipv4.device contains the network interface name. (If your host has multiple network-facing interfaces, you’ll need to modify this.)

    This PF ruleset pulls in two external files, one containing a list of management addresses and one containing the complete list of my internal networks.

    I allow access from my management networks, allow ICMP, default block, all the routine packet filter stuff. The next interesting bit is the allowed services. I check for the host’s presence in a group, and if it’s there’ I add a rule to permit the access that protocol needs.

    One detail that gave me trouble made me use inventory_hostname rather than ansible_fqdn or ansible_hostname to check group membership. I manage systems in several domains, and many of them have one name in our management systems and another in DNS. I put machines in Ansible by their fully qualified domain name. Using ansible_fqdn to get a hostname returns the hostname given in reverse DNS. inventory_hostname returns the hostname as it appears in the hosts file. If ansible_fqdn doesn’t match the hostname in the hosts file, the group comparison fails. Using inventory_hostname gave me one consistent set of hostnames for comparisons.

    So now I can easily deploy a secure NTP configuration to my servers. When I have to deploy some other service that requires updating the packet filter, I can include the same task file. And the handlers are now similarly reusable.

    Configuring the clients across several different operating systems will probably require Ansible roles, however. I’d best get on that next…

    Wanted: a VAX

    No, not for me. If I bring another piece of obsolete hardware into this house, it can have my chair because I’ll be sitting out at the curb.

    The OpenBSD Project builds all their packages on native hardware. Yes, it might take a month to build a complete package set on some of their platforms, but that’s okay.

    Their VAX recently died. They need a new one.

    I know that some of you have a VAX in storage, that you’ve been reluctant to surrender because it’s a cool toy. Now you can send it to a good home. You can tell everyone that you, personally, saved OpenBSD’s VAX support.

    How often can your pack rat tendencies make you a hero? This is your chance. Take it.