I recently learned that I could rent a dedicated machine from bloom.host for less than I’ve been paying for my virtual machines. Time to move some VMs to jails! Here’s the notes I’ve left for myself. All of my VMs run ZFS.
First, clean up unneeded boot environments, remove any unnecessary crap that lingered on the VM, apply all security updates, and in general tidy up the source VM.
Then decide how you want to flip services over. The cleanest way is to shut down all services and start the migration, but you might need to guarantee uptime. It’s up to you. I chose to leave services running during an initial replication, shut down services, do an final snapshot with an incremental replication, start the new jail, and change DNS to the new addresses. Figure out your own uptime requirements.
Start by creating a recursive snapshot of the system.
# zfs snapshot -r zroot@bloom
At a convenient time, I’d go to destination host and pull the snapshots over. The snapshots need to go into a directory on the zroot/jails dataset, named after the VM the jail will replace.
$ ssh mwlucas@www.mwl.io zfs send -Rc zroot@bloom | zfs recv -v -o mountpoint=/www zroot/jails/www
This might take a while, so follow up with an incremental right before you want the actual the migration.
$ ssh mwlucas@www.mwl.io zfs send -Rci zroot@bloom2 zroot@bloom3 | zfs recv -v -o mountpoint=/jails/mail zroot/jails/www
if you’ve tampered with new datasets between copies, you’ll get an error.
receiving incremental stream of www/ROOT@bloom3 into zroot/jails/www/ROOT@bloom3
cannot receive incremental stream: destination zroot/jails/www/ROOT has been modified
since most recent snapshot
warning: cannot send 'www/ROOT/default@bloom3': signal received
Broken pipe
Roll back the problem dataset.
# zfs rollback zroot/jails/mail/ROOT@bloom2
Data’s moved over, but there’s trouble.
$ zfs list
...
zroot/www 39.6G 776G 132K /www
zroot/www/ROOT 22.5G 776G 132K /www/ROOT
zroot/www/ROOT/default 22.5G 776G 21.8G /www/ROOT/default
zroot/www/usr 10.9G 776G 132K /www/usr
zroot/www/usr/home 9.37G 776G 384K /www/usr/home
zroot/www/usr/home/acme 7.10M 776G 7.10M /www/usr/home/acme ...
The jail boots from the boot environment /www/ROOT/default, but the jail’s root dataset is /zroot/www. It’s empty. Shuffling datasets and rearranging inheritance is a pain. I just duplicated the contents
# zfs mount zroot/jails/mail/ROOT/default
$ tar cfC - /jails/www/ROOT/default/ . | tar xvpfC - /jails/www/
# zfs list zroot/www
NAME USED AVAIL REFER MOUNTPOINT
zroot/www 41.4G 774G 132K /www
zroot/www/ROOT 22.5G 774G 132K /www/ROOT
zroot/www/usr 10.9G 774G 132K /www/usr
zroot/www/var 7.96G 774G 132K /www/var
Go into the jail’s root directory. Edit /etc/sysctl.conf to remove non-jail settings. You can also edit rc.conf for the new network interface and the new IP.
I’m using VNET, because otherwise I must configure on-system daemons to avoid binding to localhost. (Remember, in a non-VNET jail localhost is aliased to the public IP!) That means I need a bridge interface. This host has one live Ethernet, igb0 so I make it a bridge.
autobridge_interfaces="bridge0"
autobridge_bridge0="igb*"
cloned_interfaces="bridge0"
ifconfig_igb0="UP"
I then add a public IP to the bridge, for the host’s use.
Now for jail.conf for a VNET install. I need to allow devfs for running named(8) on some of the VMs, and I want raw sockets.
path = "/jails/$name";
mount.devfs;
devfs_ruleset=5;
exec.clean;
allow.mount.devfs=1;
allow.raw_sockets=1;
exec.consolelog="/jails/$name/var/log/console.log";
vnet;
exec.prestart += "/sbin/ifconfig epair${jid} create up";
exec.prestart += "/sbin/ifconfig epair${jid}a descr 'vnet-${name}'";
exec.prestart += "/sbin/ifconfig bridge0 addm epair${jid}a up";
vnet.interface="epair${jid}b";
exec.start = "sh /etc/rc";
exec.created="logger jail $name has started";
exec.stop = "sh /etc/rc.shutdown";
exec.poststop += "ifconfig epair${jid}a destroy";
exec.poststop +="logger jail $name has stopped";
.include "/etc/jail.conf.d/*.conf";
This reduces individual jail.conf entries to this.
www {
jid = 80 ;
}
At this point, I could start the jail and see what broke. Some common errors included /tmp losing the sticky bit and MariaDB directories being owned by root rather than mysql.
Change the DNS, and watch traffic shift to the new host.
Am I confident in this process? No. That’s why I make sure I have a last backup in Tarsnap, and wait 30 days to delete the source VM.