I’m building a new virtualization cloud with SolusVM, KVM, and a bit of Xen (to make use of older hardware). Each machine has its own hard disk, but it only holds the local operating system. All virtual machines reside on cheap iSCSI storage, so I can easily migrate VMs from one compute node to another. The goal being, of course, to separate service failures from hardware failures. (I still have to deal with possible storage failures, of course, but hot-swap hard drive arrays reduce my risk somewhat.)
SolusVM provides a nice front end to the whole Linux virtualization tangle. It does exactly what it claims, and at a reasonable price. I’m happy to pay someone a couple bucks a year per physical server to give me a non-sucky cloud front end that Just Works. One feature that it lacks is live migration for KVM and Xen hosts. Live VM failover is nice, but not essential for my purposes. As part of our Redundant Array of Inexpensive Crap strategy, I cluster VMs as well as physical servers: multiple mail servers, multiple DNS servers, and so on.
While there’s documentation on how to cold-migrate Xen VMs, there’s no documentation on how to migrate a KVM VM from one node to another, however. Let alone how to do this with shared storage. But the forum says that the Xen method should work with KVM. Let’s try it and see what happens!
The Xen page talks about replicating the LVM container on the new node. With shared storage, you can skip this step; I defined my SolusVM groups based on the iSCSI device they’re attached to. I imagine the same migration process would work with unshared storage, if you duplicated the disk data first.
Go into the SolusVM GUI and note the VM number and the node number. For my test, want to move VM 2 onto node 4. Log onto the master server, become root, and run:
# cd /scripts
# ./vm-migrate 2 4
Virtual server information updated!
#
I then tried to start the VM via the GUI, and it wouldn’t boot. Log onto the compute node to find out why. Any time I have a virtualization problem involving multiple pieces of hardware, I check /var/log/libvirt/libvirtd.log. Starting the virtual machine generated this log message:
14:36:13.417: 1443: error : qemuMonitorOpenUnix:290 : failed to connect to monitor socket: No such process
14:36:13.417: 1443: error : qemuProcessWaitForMonitor:1289 : internal error process exited while connecting to monitor: inet_listen_opts: bind(ipv4,0.0.0.0,5901): Address already in use
inet_listen_opts: FAILED
The KVM instance could not use port 5901, because something else was using it. KVM uses VNC to offer console access, and attaches to a port above 5900. Machine number one’s console is on VNC on port 5901, machine number two on port 5902, and so on.
The migrate-vm script didn’t change the console port. I went into the VM entry, changed the port by hand, and brought up the machine without trouble. Annoying, but not insurmountable.
Hopefully this helps the next sysadmin searching for this topic.