Category: bsd

I was researching next week’s OpenBSD talk and thought “You know, I ought to tell the story about VRRP, CARP, and Cisco. That’s a good one, and it illustrates how the OpenBSD community works and thinks.” It’s been ten years, so I decided to do some research to make sure I had my facts straight.

And I came across the Cisco Nexus 1000V manual. This big and mighty Cisco switch… supports CARP.

This is absolutely hilarious. I laughed so much my sides hurt.

Some of you younger folks are probably wondering what the big deal is. Well…

Back in the late 1990s, Cisco came up with the Virtual Router Redundancy Protocol (VRRP), using some of the lessons of their Hot Standby Router Protocol (HSRP). This was a quick-acting router failover protocol. If one router died, a second would notice and automatically take over for it. VRRP isn’t rocket surgery, it’s just that Cisco’s hardware could now support it and the market demanded it. Fair enough.

But then Cisco patented VRRP.

Cisco announced that anyone could implement VRRP, so long as they didn’t sue Cisco over it. Cisco wanted to offer something to the world, and didn’t want it to come back and bite them. Again, fair enough. Perfectly sensible from Cisco’s perspective.

The OpenBSD folks wanted router redundancy, too. And they wanted it in the base system. But Cisco’s licensing terms were a problem.

The modern BSD license boils down to:

1) Keep our copyright notice on this code
2) Don’t sue us if it breaks

There’s nothing in there about “And don’t sue Cisco if something breaks.” Specifically, the code can be used for any purpose, including suing Cisco. Mind you, you’d have a pretty hard time using OpenBSD code to sue Cisco, but the license doesn’t prohibit it.

So, while the VRRP patent terms were fine for Cisco, they weren’t acceptable under the BSD license.

And the OpenBSD devs wanted redundancy.

What to do? Go off and write your own protocol, the Common Address Redundancy Protocol (CARP). Make it different from VRRP. Field-test the protocol, using your legions of willing lackeys — er, devoted userbase. Make CARP not only a usable replacement for VRRP, but inherently better and stronger. Put the protocol under the BSD license, and give the protocol and code away.

This caused something of a kerfuffle at the time. Ugly accusations flew around. “It’s a VRRP knock-off!” “No, it’s a different protocol!” Great big reams of email were written about the whole thing.

The OpenBSD folks applied to IANA for a protocol number. IANA rejected the application, telling them to use VRRP instead. VRRP was assigned protocol 112. So OpenBSD used protocol 112 for CARP. And putting CARP hosts on a network with Cisco VRRP hosts made Cisco routers crash. The Cisco stack wasn’t robust enough to handle strange packets on the network. Cisco updated their hardware to survive seeing a lone CARP packet.

This escalated the kerfuffle into industry news. You’d see articles in all kinds of industry magazines about OpenBSD versus Cisco.

The OpenBSD folks responded by doing a CARP/VRRP-themed 3.5 release, complete with a Monty Python parody (lyrics, MP3).

And in the end of it all… everyone shut up. Other people started implementing CARP. Because it’s a solid, respectable redundancy protocol. You can get CARP from FreeBSD, Linux, Solaris, and a whole bunch of other vendors…

…including Ciso.

I had plans for today, but I’m too busy laughing. And then I need to go watch some Monty Python.

The fine folks at BSDNow.tv have requested my presence this Wednesday for an interview. We’ll talk sudo, FreeBSD, OpenBSD, and whatever else comes to mind.

You can watch it live, this Wednesday at 2PM EST.

PS: Hey, boss, I’ll be out Wednesday afternoon. Personal business.

The traditional BSD standard of “edit /etc/rc.conf” isn’t sustainable across large numbers of machines. If you must change dozens of servers you want a reliable way to alter the system without either manually editing every configuration file or some sed/awk hackery. (Running a sed/awk script to edit rc.conf on every server I own makes me nervous. I don’t do nervous these days.) FreeBSD 9.2 and later includes sysrc, a program to consistently and safely alter /etc/rc.conf and friends from the command line. (On older versions of FreeBSD, sysrc is available in ports.) I find sysrc very useful for Ansible-managed farms, but it should work just as well with Puppet or Chef or any configuration management system.

Start using sysrc by asking it what it knows about the system configuration.

$ sysrc -a
defaultrouter: 192.0.2.129
dumpdev: AUTO
hostname: mwltest3
ifconfig_em0: inet 192.0.2.160 netmask 255.255.255.128
keymap: us.dvorak.kbd
sshd_enable: YES

Oddly enough, this is exactly what’s in my rc.conf, minus all the comments and such.

I must enable ntpd(8) on this machine. Here, sysrc looks an awful lot like sysctl.

# sysrc ntpd_enable=YES
ntpd_enable: NO -> YES

Note that this doesn’t actually start ntpd, it merely enables it in the configuration. You must run /etc/rc.d/ntpd start or service ntpd start to start the daemon.

To disable a daemon, do the same thing in reverse.

# sysrc ntpd_enable=NO
ntpd_enable: YES -> NO

One potential problem with sysrc is that it’s a tool for editing /etc/rc.conf, not for configuring FreeBSD. It has no idea what legitimate values are. This means that if you make a typo, it propagates to rc.conf.

# sysrc ntpd_enable=YSE
ntpd_enable: NO -> YSE

Here sysrc works as advertised, but ntpd won’t. And you can arbitrarily enable nonexistent services.

# sysrc gerbil_enable=YES
gerbil_enable: -> YES

I do not have gerbils installed. But if I ever do install them, the gerbil wheel will start spinning without any further intervention.

I suspect that one day sysrc will grow a service integrity checker, but it solves my immediate needs.

Experienced FreeBSD administrators who run a small number of servers don’t need sysrc, but those of us with farms will find it invaluable. My thanks to Devin Teske for shepherding this into base.

My Web server runs FreeBSD. Some security updates prompted me to upgrade my installed packages. Running this server entirely with packages isn’t possible, so I installed PHP 5 from ports. This machine uses pkgng.

When I ran pkg upgrade, however, my web site stopped working. The server itself started just fine, and it did quite well serving downloadable PHP code to clients. The problem was pretty obvious:

I originally compiled PHP 5 from ports so to get the Apache PHP module. Running pkg upgrade replaced my custom-built PHP with one from packages, so Apache no longer had a PHP module.

So: I want to upgrade from packages, but not upgrade PHP automatically. pkgng has a tool for this, pkg lock. You must give pkg lock a package name and confirm that you want to update this package.

# pkg lock php5
php5-5.4.17: lock this package? [y/N]: y
Locking php5-5.4.17
#

If you give a nonexistent package name, pkg exits silently. Which is kinder than calling you an idiot, I suppose.

# pkg lock php-5
#

Personally, I’d rather be told I’m an idiot (aka “no such package”), because when I’m tired I might interpret this as “package locked, everything is good.” But whatever.

Before uninstalling a locked port, either via pkg or ports, you must unlock the package.

To upgrade this server now, I do the following:

# portsnap fetch update
# pkg upgrade
# cd /usr/ports/lang/php5 && make
# pkg unlock php5
# make deinstall && make reinstall
# pkg lock php5

Done!

I’m told by people who should know that eventually pkgng will let me do this entirely with packages, but for now, this will do nicely.

From my experience, upgrading with pkgng is MUCH nicer than upgrading with pkg_add. All of the issues that drove me away from binary package upgrades have disappeared.

I should also note, however, that the BSDs will have the ports tree for the forseeable future. We need it. Some popular ports, such as nginx, have 77,371,252,455,336,267,181,195,264 possible combinations. The packaging team is not going to build 2^86 nginx packages. But you can build whichever exact version you need.

It seems that pkgng is actually taking FreeBSD binary packaging back towards being useful. Once there’s an official public repository of pkgng packages, you really should try it out.

(Update: Hat tip to Allan Jude for pointing out that 2^86 is an impressively big number when you multiply it out.)

Ansible includes a module to manage FreeBSD packages, if you’re using the forthcoming pkgng packaging system. The Ansible module isn’t complete yet, but as Ansible is moving really quickly, I’m pretty confident their FreeBSD support will grow additional knobs. As pkgng is increasingly close to production, and the PC-BSD folks have generously offered their 64-bit pkgng repository available to the public, this seems like a good time to make the move.

But I’m not about to make this change manually. Bootstrapping pkgng isn’t difficult, but I have a great big heap of FreeBSD VMs and I have other things I’d like to accomplish this month. Therefore, I’m bootstrapping my ability to manage FreeBSD packages via Ansible, with Ansible.

Before starting, you need an Ansible server and a pkgng repo.

All of my FreeBSD servers run 9.1, updated via freebsd-update. If you come across this article years later, adjust accordingly.

My Ansible server runs OpenBSD, and the OpenBSD ansible package has problems managing anything other than OpenBSD. I generically recommend running Ansible out of git.

You also need a pkgng repository. The official repository is in closed testing, but many FreeBSD developers are using it successfully. PC-BSD has made their 64-bit repository available to all FreeBSD users. And many people have built their own repository. Thanks to my awesome Twitter stalkers minions followers, I have access to more than one private repository. This example assumes you’re using the 64-bit-only PC-BSD repository.

Configure the pkgng repo in pkg.site. I keep my FreeBSD configuration files in /home/ansible/freebsd/etc/, so I make a /home/ansible/freebsd/etc/pkg.site that contains only:

packagesite: http://pkg.cdn.pcbsd.org/9.1-RELEASE/amd64
PUBKEY: /usr/local/etc/pkg-pubkey.cert
PKG_CACHEDIR: /usr/local/tmp

I also need the current PC-BSD public key, saved as pkg-pubkey.cert

With these two files and an Ansible install, we’re ready to deploy on the Ansible group freebsd-test. Here’s the runbook.

---
- hosts: freebsd-test
  user: ansible
  sudo: yes

  tasks:
  - name: install pkg tools
    action: command  pkg_add -r pkg
#do you need a proxy? Put it here
#    environment:
#      ftp_proxy: http://proxy.michaelwlucas.com:8080

  - name: edit /etc/make.conf
    action: shell echo "WITH_PKGNG=YES" >> /etc/make.conf

  - name: convert package database
    action: shell pkg2ng

#I have typed pkg_add for 18 years, and my fingers no longer listen to
#my brain. Disable pkg_* commands for safety
  - name: disable pkg_ commands
    action: shell chmod -x /usr/sbin/pkg_*

  - name: install pkg.conf
    action: copy src=/home/ansible/freebsd/etc/pkg.conf
      dest=/usr/local/etc/pkg.conf owner=root group=wheel mode=0644

#skip this if you're using a non-PCBSD repo
  - name: install pc-bsd pgp key
    action: copy src=/home/ansible/freebsd/etc/pkg-pubkey.cert
      dest=/usr/local/etc/pkg-pubkey.cert owner=root group=wheel mode=0644

#ansible pkg does not have upgrade command yet
#use shell to trigger upgrade
#pkgng package in pkg-old is always out of date, upgrade it
  - name: upgrade pkg pkg
    action: command pkg upgrade -qy 
#do you need a proxy? Put it here
#    environment:
#      ftp_proxy: http://proxy.michaelwlucas.com:8080

This takes a while to run.

Before deploying, test. Test again. And run your conversion in batches, so that you don’t scramble several hundred virtual machines simultaneously. Because that would really suck. Fortunately, by changing the group at the top of the playbook or specifying a new inventory file, you can batch these changes easily.