CARP and devd on FreeBSD

In my last post I discussed using HAST with ZFS.  That tells you how to replicate a filesystem back and forth between two machines.  That’s nice, as far as it goes, but I want automatic failover.  Clustering.  I want to wake up in the morning to a message that says “machine 1 failed, machine 2 took over, and nobody noticed” instead of a lot of messages from angry customers.  The standard FreeBSD failover mechanism is CARP, the Common Access Redundancy Protocol.  Here’s the basics of CARP.

You should read the official FreeBSD documentation on CARP, but the next couple paragraphs might get you started.  First, build a kernel that includes device carp.  (Sadly, CARP is not available as a kernel module… if you’re looking for an interesting kernel programming project, this would be a great benefit to the community).  CARP uses a virtual IP, and moves that IP between pairs of machines as needed.  Whatever machine holds the virtual IP provides services.  If one machine fails, the backup automatically takes over the IP and offers the services.

For example, I have two test machines:  skunk1 (10.0.0.1) and skunk2 (10.0.0.2).  Interface carp0 on both machines has the address 10.0.0.3.  When the systems boot, carp0 is up on skunk1 and down on skunk2.  When skunk1 fails, skunk2 activates its carp0 interface.  This has worked quite well for firewalls and other network devices for many years.  It’s also been used for application servers, but systems administrators have needed to synchronize the files on the disks by some other means.

Configure CARP in /etc/rc.conf.  Here’s skunk1:

cloned_interfaces=”carp0″
ifconfig_carp0=”vhid 1 pass FeedLucasGelato advskew 10 10.0.0.3 netmask 255.255.255.0″

And here’s skunk2:

cloned_interfaces=”carp0″
ifconfig_carp0=”vhid 1 pass FeedLucasGelato advskew 20 10.0.0.3 netmask 255.255.255.0″

Reboot, and you should have CARP interfaces.  The host with the lower “advskew” value is the master.  I can switch a host between master and backup roles by using ifconfig(8) to raise the advskew.  For example, skunk1’s carp0 has an advskew of 10.  skunk2 has an advskew of 20.  skunk1 is master.  To switch skunk1 to backup, I would just run

# ifconfig carp0 advskew 30

Poof!  skunk2 is now in charge.

All this is old hat.  What most people don’t realize is that on FreeBSD 8, CARP interfaces generate devd up and down events.  This means we can watch for an interface state change, and take action when we see that change.  (I covered devd in Absolute FreeBSD; look there for more detail.)  Add any local devd changes to /usr/local/etc/devd/devd.conf.  Here’s my local devd.conf:

notify 30 {
match “system”          “IFNET”;
match “subsystem”       “carp0”;
match “type”            “LINK_UP”;
action “/usr/local/scripts/carp-hast-switch master”;
};

notify 30 {
match “system”          “IFNET”;
match “subsystem”       “carp0”;
match “type”            “LINK_DOWN”;
action “/usr/local/scripts/carp-hast-switch slave”;
};

The leading “notify 30” gives the type of event (notify) and priority (30).  This devd rule has a high priority, and will be processed before the system’s default rules.  The following rules identify the CARP interface we’re watching.  Finally, the action points us at a shell script to run when we are notified of the event.

We can now run arbitrary commands when our CARP interfaces go up and down.  Next time, we’ll glue devd, CARP, HAST, and ZFS together in one great big pile of yummy redundancy goodness.