convert FreeBSD to pkgng with Ansible

Ansible includes a module to manage FreeBSD packages, if you’re using the forthcoming pkgng packaging system. The Ansible module isn’t complete yet, but as Ansible is moving really quickly, I’m pretty confident their FreeBSD support will grow additional knobs. As pkgng is increasingly close to production, and the PC-BSD folks have generously offered their 64-bit pkgng repository available to the public, this seems like a good time to make the move.

But I’m not about to make this change manually. Bootstrapping pkgng isn’t difficult, but I have a great big heap of FreeBSD VMs and I have other things I’d like to accomplish this month. Therefore, I’m bootstrapping my ability to manage FreeBSD packages via Ansible, with Ansible.

Before starting, you need an Ansible server and a pkgng repo.

All of my FreeBSD servers run 9.1, updated via freebsd-update. If you come across this article years later, adjust accordingly.

My Ansible server runs OpenBSD, and the OpenBSD ansible package has problems managing anything other than OpenBSD. I generically recommend running Ansible out of git.

You also need a pkgng repository. The official repository is in closed testing, but many FreeBSD developers are using it successfully. PC-BSD has made their 64-bit repository available to all FreeBSD users. And many people have built their own repository. Thanks to my awesome Twitter stalkers minions followers, I have access to more than one private repository. This example assumes you’re using the 64-bit-only PC-BSD repository.

Configure the pkgng repo in pkg.site. I keep my FreeBSD configuration files in /home/ansible/freebsd/etc/, so I make a /home/ansible/freebsd/etc/pkg.site that contains only:

packagesite: http://pkg.cdn.pcbsd.org/9.1-RELEASE/amd64
PUBKEY: /usr/local/etc/pkg-pubkey.cert
PKG_CACHEDIR: /usr/local/tmp

I also need the current PC-BSD public key, saved as pkg-pubkey.cert

With these two files and an Ansible install, we’re ready to deploy on the Ansible group freebsd-test. Here’s the runbook.

---
- hosts: freebsd-test
  user: ansible
  sudo: yes

  tasks:
  - name: install pkg tools
    action: command  pkg_add -r pkg
#do you need a proxy? Put it here
#    environment:
#      ftp_proxy: http://proxy.michaelwlucas.com:8080

  - name: edit /etc/make.conf
    action: shell echo "WITH_PKGNG=YES" >> /etc/make.conf

  - name: convert package database
    action: shell pkg2ng

#I have typed pkg_add for 18 years, and my fingers no longer listen to
#my brain. Disable pkg_* commands for safety
  - name: disable pkg_ commands
    action: shell chmod -x /usr/sbin/pkg_*

  - name: install pkg.conf
    action: copy src=/home/ansible/freebsd/etc/pkg.conf
      dest=/usr/local/etc/pkg.conf owner=root group=wheel mode=0644

#skip this if you're using a non-PCBSD repo
  - name: install pc-bsd pgp key
    action: copy src=/home/ansible/freebsd/etc/pkg-pubkey.cert
      dest=/usr/local/etc/pkg-pubkey.cert owner=root group=wheel mode=0644

#ansible pkg does not have upgrade command yet
#use shell to trigger upgrade
#pkgng package in pkg-old is always out of date, upgrade it
  - name: upgrade pkg pkg
    action: command pkg upgrade -qy 
#do you need a proxy? Put it here
#    environment:
#      ftp_proxy: http://proxy.michaelwlucas.com:8080

This takes a while to run.

Before deploying, test. Test again. And run your conversion in batches, so that you don’t scramble several hundred virtual machines simultaneously. Because that would really suck. Fortunately, by changing the group at the top of the playbook or specifying a new inventory file, you can batch these changes easily.

“Absolute OpenBSD” auction photographs

I promised to post photographs of the signed copy of Absolute OpenBSD 2nd Edition that went to Bill Allaire.

Here they are. In full size, so that Bill can later use them to authenticate his copy, in the extraordinarily unlikely event that he desires to do so. (And also so that perhaps the bandwidth utilized might finally exceed that generated by my fame in the gay porn world.)

All the developers signed the first page:
ao2e inside page

People immediately checked the index for their favorite features. Leave it to Bob Beck to find my extraordinarily subtle joke on BUFCACHEPERCENT, which I assure you I completely deliberately chose to do and was not at all an actual error.

ao2e index

Some developers weren’t terribly confident of their English skills. But that’s okay.
ao2e japanese

Sadly, there were some problems with the book. People were kind enough to point them out.
ao2e small margin

Many pages didn’t get marked up, but a few got extra attention. We now know exactly how Theo feels about file flags.
ao2e 174-175

Finally, Theo was kind enough to add a footnote. He didn’t mark it from the main body of the text, however. Programmers might have mad skillz, but footnotes are much harder than they look.
ao2e footnote page

As an aside: I considered getting the developers to sign my personal copy of the book while I was there. But that would reduce the uniqueness of this artifact, dang it. So Bill gets the awesome copy and the rest of us get pix.

Visiting an OpenBSD hackathon.

I took Bill Allaire’s copy of Absolute OpenBSD to Toronto to get it signed. If you wonder what that’s like, check out my article over on undeadly.org.

Why no tech posts from me for a while now? I’m moving a bunch of virtual machines from ESX to SolusVM. As some of these VMs are several years old, I’m taking the opportunity to make new OS installs and get them into my Ansible setup. It’s a lot of work, but it’s not blog-worthy.

When I do something interesting, you’ll be the first to know. Well, no, maybe the sixth or seventh. But still, pretty early on.

FreeBSD-update vs bind99-base

My master nameserver runs BIND 9.9, so I can do DNSSEC easily. I’ve installed from ports, but used the REPLACE_BASE option so that it overwrites the BIND 9.8.3 install included in the base system. That way I don’t have to worry about having multiple versions of the same command on different systems.

I patch this system via freebsd-update. After applying the latest security patches, I got the following email:

The following files will be updated as part of updating to 9.1-RELEASE-p3:
/usr/bin/dig
/usr/bin/host
/usr/bin/nslookup
/usr/bin/nsupdate
/usr/sbin/ddns-confgen
/usr/sbin/dnssec-dsfromkey
/usr/sbin/dnssec-keyfromlabel
/usr/sbin/dnssec-keygen
/usr/sbin/dnssec-revoke
/usr/sbin/dnssec-settime
/usr/sbin/dnssec-signzone
/usr/sbin/lwresd
/usr/sbin/named
/usr/sbin/named-checkconf
/usr/sbin/named-checkzone
/usr/sbin/named-compilezone
/usr/sbin/named-journalprint
/usr/sbin/rndc-confgen

I don’t want freebsd-update to patch these files. I also don’t want to get an email every day telling me that I need to patch them. I know I don’t need to patch them.

The solution? Tell freebsd-update to ignore these files with the IgnorePaths directive in /etc/freebsd-update.conf. I copied the list of files from the email and added IgnorePaths before them.

...
IgnorePaths /usr/bin/dig
IgnorePaths /usr/bin/host
IgnorePaths /usr/bin/nslookup
IgnorePaths /usr/bin/nsupdate
IgnorePaths /usr/sbin/ddns-confgen
IgnorePaths /usr/sbin/dnssec-dsfromkey
IgnorePaths /usr/sbin/dnssec-keyfromlabel
IgnorePaths /usr/sbin/dnssec-keygen
IgnorePaths /usr/sbin/dnssec-revoke
IgnorePaths /usr/sbin/dnssec-settime
IgnorePaths /usr/sbin/dnssec-signzone
IgnorePaths /usr/sbin/lwresd
IgnorePaths /usr/sbin/named
IgnorePaths /usr/sbin/named-checkconf
IgnorePaths /usr/sbin/named-checkzone
IgnorePaths /usr/sbin/named-compilezone
IgnorePaths /usr/sbin/named-journalprint
IgnorePaths /usr/sbin/rndc-confgen
...

The complication here is that I must watch out for BIND security advisories, rather than just trusting in the update process. But that’s normal.

The next FreeBSD book?

BSDCan hasn’t officially started, and I keep getting asked when I will write a third edition of Absolute FreeBSD?

The short answer is: I don’t know.

The slightly longer answer is: it depends in part on you.

The much longer answer is:

FreeBSD has added lots of stuff since Absolute FreeBSD came out in 2007. The big, screaming, basic change is that ZFS is really well-supported, and considered a core feature.

But you can’t install to ZFS. Or to a mirror. Or to any of the other really cool options available on FreeBSD. There’s good stuff there, but new users can’t have it.

There are ways around this. For a new user, they range from ugly to absurd. I had hopes for the new FreeBSD installer, but none of the rumored improvements have reached real users. I could write “To install FreeBSD, install PC-BSD.” But my gut rebels. If you want PC-BSD, install PC-BSD.

If I was to write a big FreeBSD book today, it would have to be “1001 ways to install FreeBSD.” It would cover getting FreeBSD onto ZFS, or mirrors, or GELI, or any of the other cool options. That’s not a fun book to write and would not be fun to read.

Some discussions at BSDCan give me hope for an improved installation process. I don’t care if it’s the current installer, or a port of PC-BSD’s installer, or a resurrected sysinstall.

I have no problem spending a chapter on planning an installation, or on things you should know before installing — just look at Absolute OpenBSD. But that chapter can’t be “Here’s FreeBSD disk management system, so you can boot off a live filesystem and manually edit disks and create zpools and GELIs and mirrors and and and…” Experienced FreeBSD users put up with this, but you can’t give this to a new user.

FreeBSD should have a decent partitioning scheme at boot. If the OpenBSD folks can manage that with their text-only installer, surely FreeBSD could do so. But at least it’s possible to partition the disk with the current installer.

So: if you’re a programmer and want a new version of the book, work on the installer.

If you’re not a programmer: bribe one.

I have some small books on FreeBSD on my schedule, but that’s a very different thing. Until the installer changes, Absolute FreeBSD 3 isn’t even on the schedule.

“Absolute OpenBSD” Kindle edition: problems and on sale

I’ve heard from a few different readers that there are problems with the Kindle edition of Absolute OpenBSD. It’s by no means every copy in every format, but there are enough problem reports that it needs investigating.

I’ve notified the publisher. They are investigating. When I have an update, I will post it.

Generally, when a Kindle file has a problem, remove it from your device. When the new version appears, you can re-download it from Amazon.

And this is the point where I mention that Absolute OpenBSD is O’Reilly’s Deal of the Day. If you want the ebook, you can get it for 50% off.

“Absolute OpenBSD” auction winner

The final total on the Absolute OpenBSD first copy auction was: $1145. The lucky (for various interpretations of lucky) winner is Bill Allaire, long-time OpenBSD supporter.

Bill has already sent Austin the money, which is winging its way to the OpenBSD Foundation as I write this. So he’s lived up to his end of the bargain.

I’m not going to ship him the book when it’s printed, however. Us writers are flaky and untrustworthy, and it’s time people realize and accept this. For another reason, though, there’s a small OpenBSD hackathon in Toronto at the end of May. I’ve been invited to come hang out on beer night.

I’m taking Bill’s book with me, so various OpenBSD developers can sign it and point out any errors. I’m sure they’ll also offer corrections and commentary because, well, you give a BSD developer a beer and he’ll tell you what he really thinks.

So, Bill not only gets the first copy off the press. He gets the most correct version. A book more awesome than anything anyone else will have. Don’t you wish you had outbid him?

“Absolute OpenBSD” auction clears $1000

Apparently double-dog daring you people works.

The Absolute OpenBSD Foundation auction is currently at $1035.

You have another day and a half to bid. Do so.

I’m delighted. And, as promised, I’ll have an ebook sale as a result. Probably during BSDCan.

This week has been maniacal for reasons completely outside of the new Absolute OpenBSD, but I should start to catch up on all the queued email, tweets, and whatnot next week.

First barrier breached

The Absolute OpenBSD auction has been going about 10 hours now. In less than those 10 hours, the price exceeded the amount raised for the FreeBSD Foundation.

Well done.

But I bet you slackers can’t possibly double it. No, I DOUBLE DOG-DARE YOU to double it.

How I love picking a fight in a good cause.