SolusVM KVM offline migration with shared storage

I’m building a new virtualization cloud with SolusVM, KVM, and a bit of Xen (to make use of older hardware). Each machine has its own hard disk, but it only holds the local operating system. All virtual machines reside on cheap iSCSI storage, so I can easily migrate VMs from one compute node to another. The goal being, of course, to separate service failures from hardware failures. (I still have to deal with possible storage failures, of course, but hot-swap hard drive arrays reduce my risk somewhat.)

SolusVM provides a nice front end to the whole Linux virtualization tangle. It does exactly what it claims, and at a reasonable price. I’m happy to pay someone a couple bucks a year per physical server to give me a non-sucky cloud front end that Just Works. One feature that it lacks is live migration for KVM and Xen hosts. Live VM failover is nice, but not essential for my purposes. As part of our Redundant Array of Inexpensive Crap strategy, I cluster VMs as well as physical servers: multiple mail servers, multiple DNS servers, and so on.

While there’s documentation on how to cold-migrate Xen VMs, there’s no documentation on how to migrate a KVM VM from one node to another, however. Let alone how to do this with shared storage. But the forum says that the Xen method should work with KVM. Let’s try it and see what happens!

The Xen page talks about replicating the LVM container on the new node. With shared storage, you can skip this step; I defined my SolusVM groups based on the iSCSI device they’re attached to. I imagine the same migration process would work with unshared storage, if you duplicated the disk data first.

Go into the SolusVM GUI and note the VM number and the node number. For my test, want to move VM 2 onto node 4. Log onto the master server, become root, and run:

# cd /scripts
# ./vm-migrate 2 4
Virtual server information updated!
#

I then tried to start the VM via the GUI, and it wouldn’t boot. Log onto the compute node to find out why. Any time I have a virtualization problem involving multiple pieces of hardware, I check /var/log/libvirt/libvirtd.log. Starting the virtual machine generated this log message:

14:36:13.417: 1443: error : qemuMonitorOpenUnix:290 : failed to connect to monitor socket: No such process
14:36:13.417: 1443: error : qemuProcessWaitForMonitor:1289 : internal error process exited while connecting to monitor: inet_listen_opts: bind(ipv4,0.0.0.0,5901): Address already in use
inet_listen_opts: FAILED

The KVM instance could not use port 5901, because something else was using it. KVM uses VNC to offer console access, and attaches to a port above 5900. Machine number one’s console is on VNC on port 5901, machine number two on port 5902, and so on.

The migrate-vm script didn’t change the console port. I went into the VM entry, changed the port by hand, and brought up the machine without trouble. Annoying, but not insurmountable.

Hopefully this helps the next sysadmin searching for this topic.

SSH Mastery print now on Amazon

You can now buy the print SSH Mastery from Amazon. The print version isn’t yet linked to the ebook version, but that will happen within 1-3 days.

To my surprise, Amazon has not discounted the book. I don’t know why. I spoke with their author support desk, and was told that some books get discounted and some do not, according to some internal algorithm that only Amazon knows. All of my other books get discounted before publication. I assume that SSH Mastery will be discounted as well, but I have no idea when. (Yes, Amazon has an author support desk. From clicking “Call me” to resolution and hanging up the phone, total time 2 minutes 29 seconds. I am impressed.)

For those who are interested in the numbers behind the print version:

  • If you buy from OpenBSD, I make zero. The proceeds go to support OpenBSD/OpenSSH development. I am perfectly content with this.
  • If you buy from Amazon, I make about the same as I would if you bought the ebook.
  • If you buy from my CreateSpace store, I make about twice as much money as an Amazon purchase. But there’s no Amazon Prime, no free shipping when combined with other purchase, and no discount, ever.

    Some thoughts on the CreateSpace store, while I’m at it:

    CreateSpace gives me the store automatically, for free, so I link to it; otherwise, I wouldn’t bother setting it up. Purchases from here are “giving the author extra money because you want to.” And to my surprise, someone actually bought one there, so: thank you, anonymous buyer. (Note that Amazon/CreateSpace also makes more money when you buy from the CS estore.)

    In the event that people actually start buying from the CreateSpace estore, I’d probably set up a PayPal tip jar, so that those folks who want to give me extra money can do so and still get free shipping and/or discounts. It’s a step that I’ve avoided, but if people actually want overpay me for work I honestly own, who am I to argue?

  • SSH Mastery Print Here

    Today, I received a good proof of SSH Mastery. All of the errors I know of are fixed.

    I’ve ordered 200 books for the OpenBSD Project. Those books should be manufactured this weekend and delivered next week. (For the record, the OpenBSD guys have been a pleasure to deal with.)

    Amazon should have the books available in ten days or so, Barnes & Noble and other Ingram-connected bookstores a while afterwards.

    If you absolutely must have the print book now, you can order it through my CreateSpace store. Of all the ways the book is available, I make the greatest profit on books sold through the CS store. (It also charges list price, so I’m not going to push it on people. Only order from there if you want to give me extra money.)

    If you want to pay full price, you can order it from OpenBSD. I give them the books at cost, and the rest benefits OpenBSD and OpenSSH.

    Remote Web Browsing via OpenSSH and PuTTY

    I’m installing SolusVM as a virtualization management system. It lets you manage your private cloud via a Web browser, set up resellers, and so on. When you first log in, the administrative interface locks itself down so that you can only log in from one IP address, in a sort of implicit whitelist. You must explicitly add other addresses. That’s fine, even reasonable. I had three address ranges to add: my office, the headquarters, and Fearless Leader’s office. So I went into the management interface and explicitly added the headquarters’ addresses.

    And I was locked out of the management interface. Apparently the explicit whitelist permitting HQ overwrote the implicit whitelist permitting my workstation.

    I could have opened a ticket with SolusVM and admitted that I’d ignorantly locked myself out. But I don’t like interacting with vendors. I could have driven into the office, but that would involve changing out of my bathrobe. That left logging into the management workstation via a web browser from headquarters. I’m not going to talk one of my coworkers through it if I can avoid it.

    Instead, I used SSH dynamic forwarding to connect to the SolusVM head node from an IP address at headquarters.

    You can do this with an OpenSSH server and either a PuTTY or OpenSSH client. I chose to use PuTTY because that was the computer on the couch with me. I have several OpenSSH servers at headquarters.

    Open a new PuTTY session. Enter the host, username, and server port as normal. Before opening the session, go to the left-hand side of the screen and select SSH -> Tunnels. Enter a “Source port” of 9999. Near the bottom, select “Dynamic.” Now open your SSH connection.

    You now have a SOCKS proxy running on your computer. All traffic sent to port 9999 is sent over your SSH session. Your SSH server connects you to the Internet.

    Go to your Web browser’s connection settings. In firefox, it’s Tools->Options->Advanced. Select the Network tab, then Settings. Select Manual proxy settings, then enter a SOCKS host of 127.0.0.1 port 9999. Select the SOCKS5 button. Exit the menus, hitting OK all the way back.

    Now your Web browser connects to the Internet via the SOCKS proxy running on your computer. You’re browsing the Web from the IP address of your SSH server.

    This is much faster than remote browsing options such as Remote Desktop or forwarding X11 over SSH. And it let me log into my SolusVM console without having to communicate with another human being, so everybody wins.

    Of course, you could learn about this sort of trick and more in my new SSH book.

    Print pre-order for “SSH Mastery” now available

    The OpenBSD folks have made pre-orders available for SSH Mastery.

    I’m giving the books to the OpenBSD Project at cost, and they’re selling them at list as a fund-raiser. I make nothing on books ordered through OpenBSD, but the money will go to further OpenBSD, OpenSSH, PF, and who knows what else. I’m going to imagine that the money will support the annual developer barbeque, as buying beer is more fun than buying hardware.

    I will make a decent profit on books sold through Amazon, Barnes & Noble, and brick-and-mortar stores. (Yes, I can get books into brick-and-mortar stores, with a bit of luck, a smidgeon of effort, and just a few ounces of blackmail.) I figure it’ll even out; the general public will order from Amazon.

    When will the books arrive? Once the proofs are approved, I’ll order the OpenBSD books before I approve sale through Amazon. I don’t know when that is yet, but believe me, I want this book out there.

    So, if I make nothing on this, why do it? It’s a good way to support a widely-used project that has written software that makes my job easier. So order yours now!

    Plus, I write about open-source software projects, preferably BSD-based ones. This means that I have to maintain good relationships with those projects. Not only am I a geek, I work at home. My usual social interactions consist of ignoring the threatening emails from the boss, the pet rats asking for treats, and the nagging phone calls from the social worker asking if I’ve taken my meds today. As you might guess, maintaining good relationships is not my strong point. If you have trouble keeping friends, try bribery. I hear it works a treat.

    “SSH Mastery” print status

    Thursday, I received the first print proofs of SSH Mastery.

    Initial impressions: the print version looks good. The photo doesn’t do the nice glossy cover and bright colors justice, but:

    After careful inspection, though, I found a few errors. The interior images are not as sharp as I would like. Some of these I can’t really help: printouts of screenshots look like crap. But I’m more concerned about pixellation in diagrams and images.

    Plus, there’s an extraneous colon on the back cover.

    If I want this book to be taken seriously, it needs to look like a real book. So I’m getting the layout corrected and running another round of proofs. I suspect I’ll get the new proofs late next week. If those are OK, I will be able to open orders.

    I’ve been working with Austin from the OpenBSD group to get their ordering up soon.

    January “SSH Mastery” sales, by the numbers

    I’ve promised several authors to share the results of my self-publishing experiment with SSH Mastery. I don’t have complete data yet, but I do have sales numbers for January from Amazon, B&N, and Smashwords.

    Some caveats here:

  • This includes only “SSH Mastery.” I have removed my fiction from the totals. (Fiction sales are considerably lower, but growing.)
  • February’s initial results are much lower than January’s. January’s sales are obviously to my “hard-core fans” and people close to the community.
  • My expenses aren’t yet totaled, as the print version isn’t available yet. I expect them to come in at roughly $3000. The majority of this expense is the class I took to learn how to self-pub properly, which is a one-time expenditure. A business person would argue that this expense doesn’t count, as it should be spread across multiple books. But as an author running a test project, I have to count this expense somewhere, so it’s attached to this book. Also, these expenses are only cash out of pocket; they assume my time is free.
  • On a related note, an accountant would probably find my reasoning naive. I already have two careers (writer and engineer), I’m not going to add a third.
  • I’m not going to regularly report sales numbers to the general public. I will say when I break even.
  • Before I published, I expected to make most of my sales through Amazon, then Barnes & Noble, and then Smashwords last. I had no idea of how many sales I would make, but I figured they’d be in that order. Let’s see how this compared to reality.

  • Amazon Kindle: 123 books sold (92 US, 12 UK, 14 DE, 2 FR, 1 IT, 2 ES), for a total of $810.27 USD at today’s exchange rates. (Amazon reports European royalties in euros or pounds.)
  • Barnes & Noble: 4 books sold, for $25.96 USD.
  • Smashwords: 76 books sold, for $607.65
  • Total: 202 books sold, for a total of $1443.88
  • Some interesting things here:

    I’m shocked at how low Barnes & Noble sales are. The book was available on B&N a couple days after Smashwords and Amazon, mainly because getting the book through B&N’s internal systems took longer. Apparently my readers don’t use the Nook.

    Smashwords sales as a proportion of total sales is much higher than I expected.

    Averaging the royalty per book is also interesting.

    Amazon: $6.58/sale
    Smashwords: $7.99/sale
    B&N: $6.49/sale

    You’ll hear lots of commentary about how Amazon offers a 70% royalty on ebooks. This has all sorts of exceptions and exclusions, where you’ll only get a 35% royalty. My effective average royalty at Amazon is about 65%, so that’s not bad.

    Barnes & Noble, at just under 65%, offers the worst net royalty.

    Smashwords: I love you. Just saying. My core audience really wants the book as PDF, and Smashwords offers PDF, epub, mobi, and all the other big ebook formats with one purchase. And they pay me the highest per-sale royalty. I still haven’t gotten the book through Meatgrinder into the other channels they feed, such as iBooks and Kobo. Once that happens, I expect to see their percentage rise. I wouldn’t be surprised if they overtake Amazon.

    The real lesson is: sell your work through all available channels. You can’t tell who will buy what from where.

    Installing WHMCS on FreeBSD 9.0-RELEASE

    Or, if you prefer: “WHMCS versus PHP.” Blogged for the next sysadmin searching Google.

    $DAYJOB recently acquired WHMCS to help automate virtual server provisioning, billing, and so on. According to everything I’ve read, WHMCS runs just fine on FreeBSD, so I installed the prerequisites on a 9.0-i386 machine. As with any server for PHP-based Web sites, I verified that the server processed PHP with a simple phpinfo() page. I then grabbed the WHMCS tarball (no link, you must be a customer, sorry), extracted it into the directory, ran the setup program, fed in the database information and license key…

    ..and it wouldn’t run. Calling up the app resulted in a blank page. WHMCS provides troubleshooting instructions for this exact circumstance. I enabled the requested debugging, but couldn’t get WHMCS to produce an error. Adding a bogus argument to my phpinfo() test page made an error appear, so I was confident the failure to display an error message wasn’t a server configuration problem.

    This comes down to FreeBSD’s PHP packaging.

    When you install PHP on the popular varieties of Linux, you generally get a whole slew of PHP extensions with it. BSD-based systems only install exactly what you ask for: if you want PHP but don’t request any extensions, you won’t get any extensions.

    I agree with this approach. Every piece of installed software needs patching and updating. Every piece of installed software is a potential attack vector. If I don’t need a piece of software, I don’t want it on my server.

    WHMCS doesn’t list all of the required extensions. They assume you have a kitchen-sink PHP install. After some reading and research, I found that WHMCS runs fine with the following PHP modules and extensions installed. I’ve included the version numbers for reference, but you should be able to just pkg_add -r all of these by name.

    php5-5.3.8
    php5-bz2-5.3.8
    php5-ctype-5.3.8
    php5-curl-5.3.8
    php5-dom-5.3.8
    php5-extensions-1.6
    php5-filter-5.3.8
    php5-gd-5.3.8
    php5-hash-5.3.8
    php5-iconv-5.3.8
    php5-json-5.3.8
    php5-ldap-5.3.8
    php5-mysql-5.3.8
    php5-openssl-5.3.8
    php5-pdo-5.3.8
    php5-pdo_sqlite-5.3.8
    php5-phar-5.3.8
    php5-posix-5.3.8
    php5-session-5.3.8
    php5-simplexml-5.3.8
    php5-tokenizer-5.3.8
    php5-xml-5.3.8
    php5-xmlreader-5.3.8
    php5-xmlwriter-5.3.8
    php5-zip-5.3.8
    php5-zlib-5.3.8

    I’d really like to trim this down to only what is strictly necessary to run WHMCS, but that information doesn’t seem to be available. I could methodically remove and reinstall extensions to see when WHMCS breaks, but I have better things to do than debug missing docs for a commercial PHP app.

    On the plus side: now that WHMCS is installed, it’s really slick. I’m looking forward to using it. Actually, I’m looking forward to having other people use it for me, so I can do more interesting things than provision servers, accounts, and billing.

    Basic DNSSEC with BIND 9.9

    Everybody knows that DNS is insecure, and DNS Security Extensions (DNSSEC) is supposed to fix that. I know that several of my readers consider DNSSEC suboptimal, but it’s the standard, so we get to live with it. I recently got DNSSEC working on BIND 9.9. As I write this 9.9 is in Release Candidate state, but the functionality should be basically unchanged. My goals for DNSSEC on BIND were to manually edit my zone files, but have the DNS server maintain the keys. BIND 9.9 makes this possible.

    This is a limited example of how to get basic DNSSEC working. To use it, your registrar must support DNSSEC. There’s ways around this, such as DLV, but they’re out of scope for this document. Also note that I’m not covering key rotation. That’ll be a future post.

    You also must have a domain whose parent is signed. The root zone, .com, .net, and .org are all signed, but not all top-level domains are signed. Verify your particular TLD before proceeding. Again, you can use DLV for these orphaned domains, but that’s out of scope for this document.

    I’d also suggest that you read the BIND 9.9 ARM first. But if you were going to bother to do that, you wouldn’t have done the Google search to find this article.

    You will almost certainly have service interruptions as you learn DNSSEC. I strongly recommend that you set up a test server for your DNSSEC testing. Move a test domain to it. You cannot test DNSSEC on a private domain; it must be a real, Internet-facing domain. Configure dnssec validation on this test server.

    You also need a server that provides DNSSEC resolution, but will not be authoritative for your test domain. I’m assuming that you configure DNSSEC resolution on your production server. If you only have one DNS server, you can use an offsite public resolver such as unbound.odvr.dns-oarc.net. (Note that Google DNS, like most public DNS servers, does not validate DNSSEC.)

    Verify that DNSSEC resolution works on both servers with dig(1).

    $ dig www.isc.org +dnssec

    ; <<>> DiG 9.8.1-P1 <<>> www.isc.org +dnssec
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28734 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 5, ADDITIONAL: 13

    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags: do; udp: 4096
    ;; QUESTION SECTION:
    ;www.isc.org. IN A

    ;; ANSWER SECTION:
    www.isc.org. 600 IN A 149.20.64.42
    www.isc.org. 600 IN RRSIG A 5 3 600 20120305233238 20120204233238 21693 isc.org. IKekIJVV99bkTYw4L2KG/xZpQ+BYlCK0IDSsWXKZRD8ceR/VNcfNFxV2 5VK51Fqmy...
    ...

    Two interesting things here. First, the ad flag indicates that this is “authenticated data,” also known as DNSSEC-validated. Second, the RRSIG (Resource Record Signature) is the actual DNSSEC signature. isc.org is DNSSEC-validated.

    DNSSEC generates a lot of key files. You don’t edit these key files by hand, and you rarely look at their contents, so use a separate directory. If you have a lot of zones, you’ll want a separate directory for each zone.

    You’ll need a directory for keys.

    $ mkdir /etc/namedb/keys

    To start you need a “master” key to sign other keys with (the Key Signing Key, or KSK), and then a key for each zone (the Zone Signing Key, or ZSK). Your nameserver must be able to read these keys.

    # dnssec-keygen -f KSK -a RSASHA1 -b 2048 -n ZONE example.net
    Generating key pair…………………………………………………………………………………………………………………………………………….+++ ……………………..+++
    Kexample.net.+005+38287
    # dnssec-keygen -a RSASHA1 -b 2048 -n ZONE example.net
    Generating key pair………….+++ ..+++
    Kexample.net.+005+55896
    # chown bind:bind *

    We’ve generated two key files: Kexample.net.+005+38287 (the KSK) and Kexample.net.+005+55896 (the ZSK).

    Now that you have keys, let’s look at configuring named and the zone itself.

    I recommend you enable DNSSEC logging in named.conf. If you have trouble, the DNSSEC log will identify the problem. (Actually Understanding the log is left as an exercise for the reader, the ARM, and their favorite search engine.) Make a separate directory for the log.

    # mkdir /etc/namedb/log
    # chown bind:bind /etc/namedb/log

    Then add a logging stanza in named.conf. With this configuration, the log file will never grow larger than 20Mb.

    logging {
    channel dnssec_log {
    file "log/dnssec" size 20m;
    print-time yes;
    print-category yes;
    print-severity yes;
    severity debug 3;
    };
    category dnssec {
    dnssec_log;
    };
    };

    Now set up a zone. Here I add DNSSEC data to my test domain.

    zone example.net {
    type master;
    file "master/example.net";
    key-directory "keys/";
    inline-signing yes;
    auto-dnssec maintain;
    };

    Reload your nameserver. You’ll now see the following files in the zone file directory:

    example.net
    example.net.jbk
    example.net.signed
    example.net.jnl

    Inline signing works by taking the zone file you manually maintain, transforming it into a dynamic zone, and signing the dynamic zone. DNSSEC changes are made to the journal file. As a result of this, the serial number shown to the world can differ from the serial number in your file. That’s a minor change that I’m perfectly happy to live with.

    You should now see RRSIG records in your test zone. You will not see the AD flag, however. You never see an AD flag for a zone on its authoritative nameserver.

    So, how do you test DNSSEC on your domain? You might try your second nameserver. It won’t show the AD flag either, but it should also show the RRSIG records.

    DNSSEC works via a chain of digital signatures. The root zone is signed, and your server knows about that signature. Most delegations beneath root are also signed. Your parent zone doesn’t know to trust your KSK until you tell it. This is where your registrar comes in. Create a delegation signature key (DSKEY) from your KSK.

    # dnssec-dsfromkey Kexample.net.+005+38287
    example.net. IN DS 38287 5 1 E8C01C990ACC8CEDF48379EDF9EDAB5389A9CB4E
    example.net. IN DS 38287 5 2 57EC9364CEAE50B17C0C251950B4E5B8870F6A479A94C3A92359A623 39703D53

    Copy these two lines and paste them into your registrar’s DSKEY interface. Your registrar might take one or both types of DSKEY records. I found that GoDaddy took both, but I had to remove the space from the SHA-256 (second) record.

    When your registrar updates the TLD’s zone, DNS servers that are not your authoritative zone will return the DS flag. You’ll have functioning DNSSEC.

    (Thanks to Jeffry A. Spain for his invaluable hints in debugging my first DNSSEC setup.)