Cloning a FreeBSD/ZFS Machine with ‘zfs send’

My employer’s mail server runs DirectAdmin on FreeBSD, with ZFS. The mail server is important to the company. We want to be able to restore it quickly and easily. While we back it up regularly, having a “known good” base system with all the software installed, where we can restore the mail spools and account information, would be good.

As it runs ZFS, let’s send the filesystems across the network to a blank machine.

First I need an installation ISO for the same release and architecture of FreeBSD. (Could I use a different release? Perhaps. But if I have any errors, the first thing to try will be the correct installation media. Let’s skip the preliminaries.)

I also need a web server, to store a user’s SSH public key file.

I provision a new virtual machine with exactly the same amount of disk, memory, and processor as the original.

Boot the install disk. Choose “Live CD” rather than install. Log in as root (no password).

For FreeBSD 9.2 or earlier, take care of bug 168314. The live CD can’t configure resolv.conf out of the box because the directory /tmp/bsdinstall_etc isn’t present and /etc/resolv.conf is a symlink to /tmp/bsdinstall_etc/resolv.conf.

# mkdir /tmp/bsdinstall_etc
# touch /tmp/bsdinstall_etc/resolv.conf

My disk is /dev/vtbd0. I format this disk exactly like the original server. The original install was scripted, which saves me the trouble of copying the partitioning from the original machine.

# gpart destroy -F vtbd0
# gpart create -s gpt vtbd0
# gpart add -s 222 -a 4k -t freebsd-boot -l boot0 vtbd0
# gpart add -s 8g -a 4k -t freebsd-swap -l swap0 vtbd0
# gpart add -a 4k -t freebsd-zfs -l disk0 vtbd0
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 vtbd0
# gnop create -S 4096 /dev/gpt/disk0
# kldload zfs
# zpool create -f -o altroot=/mnt -O canmount=off -m none zroot /dev/gpt/disk0.nop
# zfs set checksum=fletcher4 zroot

Our destination filesystem is all set. But we’ll need an SSH daemon to receive connections. The only writable directories are /tmp and /var, so I create a /tmp/sshd_config that contains:

HostKey /tmp/ssh_host_ecdsa_key
PermitRootLogin yes
AuthorizedKeysFile /tmp/%u

I need a host key. ssh-keygen has a -A flag to automatically create host keys, but it tries to put them in /etc/ssh. If you want to create keys in another location, you must create them manually.

# ssh-keygen -f ssh_host_ecdsa_key -N '' -t ecdsa

Now start sshd and the network.

# /usr/sbin/sshd -f /tmp/sshd_config
# dhclient vtnet0

Your original machine should now be able to ping and SSH to the blank host. The blank host should be able to ping to your web server.

This is also the point where you realize that the DHCP configured address is actually in use by another machine, and you need to reboot and start over.

Copy an authorized_keys file from your web server to /tmp/root. Also set the permissions on /tmp so that sshd will accept that key file. (The sticky bit will make sshd reject the key file.)

# fetch http://webserver/authorized_keys .
# mv authorized_keys root
# chmod 755 /tmp

At this point, the original machine should be able to SSH into the target machine as root. This was the hard part. Now recursively copy the ZFS snapshots across the network.

# zfs snapshot -r zroot@backup
zfs send -R zroot@backup | ssh root@newhost zfs recv -F zroot

Now I walk away and let gigs and gigs of data flow across the WAN. To check progress, go to the target machine and run:

# zfs list -t snapshot

When an individual snapshot finishes, it’ll appear on the list.

Once the ZFS send completes, you can reboot and have a cloned system, right?

Uh, not quite. (I suspect that all of the below could have been done before the previous reboot, but I didn’t know I had to do it. Readers, please do the below before your first reboot and let me know if it works or not.)

Reboot at this stage and you’ll get messages like:

Can't find /boot/zfsloader

Can't find /boot/kernel

What gives? For one thing, there’s no boot loader yet. I could have done this earlier, but I didn’t think of it. Boot back into the live CD and install the necessary loaders.

# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 vtbd0
bootcode written to vtbd0

While you’re in the live CD, check the ZFS pool.

# zpool status
no pools available

Hang on… I know there’s a pool here! I just saw it before rebooting. Running zpool import shows my zroot pool as an option, so let’s try it.

# zpool import zroot
cannot import 'zroot': pool may be in use from other system
use '-f' to import anyway

We know this pool isn’t in use. Let’s forcibly import it. As I’m booting off the live CD, I temporarily mount this at /mnt.

# zpool import -o altroot=/mnt -f zroot

Now that you can talk to the pool, you can tell FreeBSD which ZFS it should boot from.

# zpool set bootfs=zroot/DEFAULT/root zroot

I want to be sure this pool can import and export cleanly, so I do:

# zpool export zroot
# zpool import zroot

At this point, I’ve mounted the ZFS filesystems over the live CD filesystem. Don’t mess around with it any more, just reboot.

And your system is cloned.

Now I get to figure out how to update this copy with incremental snapshots. But that’ll be a separate blog post.