Category: bsd

The fine folks at BSDNow.tv have requested my presence this Wednesday for an interview. We’ll talk sudo, FreeBSD, OpenBSD, and whatever else comes to mind.

You can watch it live, this Wednesday at 2PM EST.

PS: Hey, boss, I’ll be out Wednesday afternoon. Personal business.

The traditional BSD standard of “edit /etc/rc.conf” isn’t sustainable across large numbers of machines. If you must change dozens of servers you want a reliable way to alter the system without either manually editing every configuration file or some sed/awk hackery. (Running a sed/awk script to edit rc.conf on every server I own makes me nervous. I don’t do nervous these days.) FreeBSD 9.2 and later includes sysrc, a program to consistently and safely alter /etc/rc.conf and friends from the command line. (On older versions of FreeBSD, sysrc is available in ports.) I find sysrc very useful for Ansible-managed farms, but it should work just as well with Puppet or Chef or any configuration management system.

Start using sysrc by asking it what it knows about the system configuration.

$ sysrc -a
defaultrouter: 192.0.2.129
dumpdev: AUTO
hostname: mwltest3
ifconfig_em0: inet 192.0.2.160 netmask 255.255.255.128
keymap: us.dvorak.kbd
sshd_enable: YES

Oddly enough, this is exactly what’s in my rc.conf, minus all the comments and such.

I must enable ntpd(8) on this machine. Here, sysrc looks an awful lot like sysctl.

# sysrc ntpd_enable=YES
ntpd_enable: NO -> YES

Note that this doesn’t actually start ntpd, it merely enables it in the configuration. You must run /etc/rc.d/ntpd start or service ntpd start to start the daemon.

To disable a daemon, do the same thing in reverse.

# sysrc ntpd_enable=NO
ntpd_enable: YES -> NO

One potential problem with sysrc is that it’s a tool for editing /etc/rc.conf, not for configuring FreeBSD. It has no idea what legitimate values are. This means that if you make a typo, it propagates to rc.conf.

# sysrc ntpd_enable=YSE
ntpd_enable: NO -> YSE

Here sysrc works as advertised, but ntpd won’t. And you can arbitrarily enable nonexistent services.

# sysrc gerbil_enable=YES
gerbil_enable: -> YES

I do not have gerbils installed. But if I ever do install them, the gerbil wheel will start spinning without any further intervention.

I suspect that one day sysrc will grow a service integrity checker, but it solves my immediate needs.

Experienced FreeBSD administrators who run a small number of servers don’t need sysrc, but those of us with farms will find it invaluable. My thanks to Devin Teske for shepherding this into base.

My Web server runs FreeBSD. Some security updates prompted me to upgrade my installed packages. Running this server entirely with packages isn’t possible, so I installed PHP 5 from ports. This machine uses pkgng.

When I ran pkg upgrade, however, my web site stopped working. The server itself started just fine, and it did quite well serving downloadable PHP code to clients. The problem was pretty obvious:

I originally compiled PHP 5 from ports so to get the Apache PHP module. Running pkg upgrade replaced my custom-built PHP with one from packages, so Apache no longer had a PHP module.

So: I want to upgrade from packages, but not upgrade PHP automatically. pkgng has a tool for this, pkg lock. You must give pkg lock a package name and confirm that you want to update this package.

# pkg lock php5
php5-5.4.17: lock this package? [y/N]: y
Locking php5-5.4.17
#

If you give a nonexistent package name, pkg exits silently. Which is kinder than calling you an idiot, I suppose.

# pkg lock php-5
#

Personally, I’d rather be told I’m an idiot (aka “no such package”), because when I’m tired I might interpret this as “package locked, everything is good.” But whatever.

Before uninstalling a locked port, either via pkg or ports, you must unlock the package.

To upgrade this server now, I do the following:

# portsnap fetch update
# pkg upgrade
# cd /usr/ports/lang/php5 && make
# pkg unlock php5
# make deinstall && make reinstall
# pkg lock php5

Done!

I’m told by people who should know that eventually pkgng will let me do this entirely with packages, but for now, this will do nicely.

From my experience, upgrading with pkgng is MUCH nicer than upgrading with pkg_add. All of the issues that drove me away from binary package upgrades have disappeared.

I should also note, however, that the BSDs will have the ports tree for the forseeable future. We need it. Some popular ports, such as nginx, have 77,371,252,455,336,267,181,195,264 possible combinations. The packaging team is not going to build 2^86 nginx packages. But you can build whichever exact version you need.

It seems that pkgng is actually taking FreeBSD binary packaging back towards being useful. Once there’s an official public repository of pkgng packages, you really should try it out.

(Update: Hat tip to Allan Jude for pointing out that 2^86 is an impressively big number when you multiply it out.)

Ansible includes a module to manage FreeBSD packages, if you’re using the forthcoming pkgng packaging system. The Ansible module isn’t complete yet, but as Ansible is moving really quickly, I’m pretty confident their FreeBSD support will grow additional knobs. As pkgng is increasingly close to production, and the PC-BSD folks have generously offered their 64-bit pkgng repository available to the public, this seems like a good time to make the move.

But I’m not about to make this change manually. Bootstrapping pkgng isn’t difficult, but I have a great big heap of FreeBSD VMs and I have other things I’d like to accomplish this month. Therefore, I’m bootstrapping my ability to manage FreeBSD packages via Ansible, with Ansible.

Before starting, you need an Ansible server and a pkgng repo.

All of my FreeBSD servers run 9.1, updated via freebsd-update. If you come across this article years later, adjust accordingly.

My Ansible server runs OpenBSD, and the OpenBSD ansible package has problems managing anything other than OpenBSD. I generically recommend running Ansible out of git.

You also need a pkgng repository. The official repository is in closed testing, but many FreeBSD developers are using it successfully. PC-BSD has made their 64-bit repository available to all FreeBSD users. And many people have built their own repository. Thanks to my awesome Twitter stalkers minions followers, I have access to more than one private repository. This example assumes you’re using the 64-bit-only PC-BSD repository.

Configure the pkgng repo in pkg.site. I keep my FreeBSD configuration files in /home/ansible/freebsd/etc/, so I make a /home/ansible/freebsd/etc/pkg.site that contains only:

packagesite: http://pkg.cdn.pcbsd.org/9.1-RELEASE/amd64
PUBKEY: /usr/local/etc/pkg-pubkey.cert
PKG_CACHEDIR: /usr/local/tmp

I also need the current PC-BSD public key, saved as pkg-pubkey.cert

With these two files and an Ansible install, we’re ready to deploy on the Ansible group freebsd-test. Here’s the runbook.

---
- hosts: freebsd-test
  user: ansible
  sudo: yes

  tasks:
  - name: install pkg tools
    action: command  pkg_add -r pkg
#do you need a proxy? Put it here
#    environment:
#      ftp_proxy: http://proxy.michaelwlucas.com:8080

  - name: edit /etc/make.conf
    action: shell echo "WITH_PKGNG=YES" >> /etc/make.conf

  - name: convert package database
    action: shell pkg2ng

#I have typed pkg_add for 18 years, and my fingers no longer listen to
#my brain. Disable pkg_* commands for safety
  - name: disable pkg_ commands
    action: shell chmod -x /usr/sbin/pkg_*

  - name: install pkg.conf
    action: copy src=/home/ansible/freebsd/etc/pkg.conf
      dest=/usr/local/etc/pkg.conf owner=root group=wheel mode=0644

#skip this if you're using a non-PCBSD repo
  - name: install pc-bsd pgp key
    action: copy src=/home/ansible/freebsd/etc/pkg-pubkey.cert
      dest=/usr/local/etc/pkg-pubkey.cert owner=root group=wheel mode=0644

#ansible pkg does not have upgrade command yet
#use shell to trigger upgrade
#pkgng package in pkg-old is always out of date, upgrade it
  - name: upgrade pkg pkg
    action: command pkg upgrade -qy 
#do you need a proxy? Put it here
#    environment:
#      ftp_proxy: http://proxy.michaelwlucas.com:8080

This takes a while to run.

Before deploying, test. Test again. And run your conversion in batches, so that you don’t scramble several hundred virtual machines simultaneously. Because that would really suck. Fortunately, by changing the group at the top of the playbook or specifying a new inventory file, you can batch these changes easily.