Moving mailboxes from Courier/Maildir to DirectAdmin/dovecot/Maildir

I have an old mail server running Postfix and courier-imap. We want to split our customers off onto their old server, preferably something with a pretty pointy-clicky interface so that they can manage their own accounts. (Yes, people do still buy email service these days.)

The old server runs FreeBSD, postfix, and courier-imap. The new server runs FreeBSD 9.2 with DirectAdmin. DirectAdmin is a canned package that puts a customer-friendly front end on a whole bunch of standard Unix software. (Why FreeBSD? ZFS. When my customers fill up the hard disk, I can just “zfs send” the whole machine to bigger disk.)

The good news is, both my old server and the new one use Maildir. The bad news is, they’re arranged slightly differently and have somewhat different file formats. Tools exist to do 80% of the migration for you, but my setup has a few edges.

I suspect that nobody else has my exact legacy setup, and my readers don’t care about this level of detail. I also suspect that the migration project will last weeks and I will completely forget how I did it in between individual customer migrations, so I’m writing this anyway. But maybe someone else can learn something from it, even if it’s just “man, I would never hire this Lucas character.”

Create your email domain and all the user accounts in DirectAdmin. I don’t know any way to migrate encrypted virtual user passwords from Postfix into DirectAdmin, so I’ll need to create new passwords for the users. That’s not uncommon for a packaged mail server migration, so my users will live with it. This is a tedious point-and-click operation, but point-and-click is the point of DirectAdmin, so give the job to a minion and get it done.

But how to transparently migrate the mailboxes from the Courier server to Dovecot?

On the old server, the mailboxes are in /disk2/mail/vhosts/domainname/account/. On DirectAdmin, the mailboxes are in /home/userid/imap/domain/account/Maildir/. I can tar up the old accounts and drag the whole directory over to the new server, but I need to massage the directories so that they’re arranged properly.

Courier and Dovecot both use files to store mailbox state information. IMAP state is important. Without it, users will re-download every single message. I don’t care about the bandwidth or the disk I/O this will cause, but the users will complain. Complaints lead to meetings, and meetings lead to the Dark Side. There’s a script to convert from one to the other. Dovecot also has good migration documentation, I recommend you read that before doing your own migration.

The file ownership needs correcting. DirectAdmin creates user accounts for each customer (more or less), and that account needs to own the email files for the various domains.

So, what am I willing to do manually?

I will manually tar up the domain’s mail directory, copy it to the destination server, and untar it in /home/userid/imap/domain/. And I’m willing to provide the username that should own the files. I’m sure both of these could be automated, but I don’t have enough accounts to migrate to do this correctly.

I’m not willing to move files around or run migration scripts. Because these are things that human beings mess up.

There’s ways to do this in shell scripts, but I’m using Perl. Because while my Perl makes small children cry, my shell scripts feature in rituals praising Nyarlathotep.

So, the script:

#!/usr/bin/perl
#migrate from old mail server to new
#run from domain directory
#Takes one argument, the username expected to own the files when done.

unless ($ARGV[0]) {
die "\nNeed a username to own files!\n\n";
}

opendir (USERS, ".") || die "Cannot open current directory";

foreach $user (readdir (USERS)) {

next if ($user =~ /^\.{1,2}$/);
print "Converting $user...\n";
mkdir ("$user/Maildir") || die "Cannot make Maildir";
opendir (USERDIR, "$user") || die "Cannot open user directory";
foreach $file (readdir (USERDIR)) {
next if ($file =~ /^\.{1,2}$|Maildir/);
print "Moving $user/$file\n";
rename ("$user/$file","$user/Maildir/$file")
|| die "Cannot move $file";
}
#next user
}
print "Now performing courier-dovecot migration...\n";
system ("courier-dovecot-migrate.pl --to-dovecot --recursive --convert");
print "Fixing ownership...\n";
system ("chown -R $ARGV[0]:mail .");

Is this pretty trivial? Yep. But the most error-prone part of any process is the part I do. The more of the migration the machine does, the fewer screw-ups in the migration.

mug.org OpenBSD talk on-line

Last night’s talk on OpenBSD is now live in the mug.org channel.

Part 1
Part 2

UPDATE: All in one.

Among other things, I compare OpenBSD to Richard Stallman and physically assault an audience member. (Brian was a very good spots, and learned an important lesson about volunteering, e.g.: don’t.) We also talk long long time, memory randomization, PF, BSD license versus GPL, Microsoft, and other OpenBSD stuff.

And, of course, the importance of the VAX.

Tonight, at mug.org

I’m planning on doing a Google hangout of my talk tonight at mug.org.

Don’t know what time I’ll be going on, however. My guess is some time a little before 7PM. Come by my Google Plus page if you want to see it live.

With luck, it’ll record as well, so you can watch it later.

Sudo Mastery print now at Amazon

You can now get a print Sudo Mastery from Amazon.

I have signed all of the Mastery books up for the Matchbook program. People who buy the print book from Amazon will soon be able to get the Kindle version for $2.99. It’s not an ideal print/ebook combo, but I’m not nearly well enough organized to ship out physical books directly.

Cisco supports CARP? Ha ha ha hahaha…

I was researching next week’s OpenBSD talk and thought “You know, I ought to tell the story about VRRP, CARP, and Cisco. That’s a good one, and it illustrates how the OpenBSD community works and thinks.” It’s been ten years, so I decided to do some research to make sure I had my facts straight.

And I came across the Cisco Nexus 1000V manual. This big and mighty Cisco switch… supports CARP.

This is absolutely hilarious. I laughed so much my sides hurt.

Some of you younger folks are probably wondering what the big deal is. Well…

Back in the late 1990s, Cisco came up with the Virtual Router Redundancy Protocol (VRRP), using some of the lessons of their Hot Standby Router Protocol (HSRP). This was a quick-acting router failover protocol. If one router died, a second would notice and automatically take over for it. VRRP isn’t rocket surgery, it’s just that Cisco’s hardware could now support it and the market demanded it. Fair enough.

But then Cisco patented VRRP.

Cisco announced that anyone could implement VRRP, so long as they didn’t sue Cisco over it. Cisco wanted to offer something to the world, and didn’t want it to come back and bite them. Again, fair enough. Perfectly sensible from Cisco’s perspective.

The OpenBSD folks wanted router redundancy, too. And they wanted it in the base system. But Cisco’s licensing terms were a problem.

The modern BSD license boils down to:

1) Keep our copyright notice on this code
2) Don’t sue us if it breaks

There’s nothing in there about “And don’t sue Cisco if something breaks.” Specifically, the code can be used for any purpose, including suing Cisco. Mind you, you’d have a pretty hard time using OpenBSD code to sue Cisco, but the license doesn’t prohibit it.

So, while the VRRP patent terms were fine for Cisco, they weren’t acceptable under the BSD license.

And the OpenBSD devs wanted redundancy.

What to do? Go off and write your own protocol, the Common Address Redundancy Protocol (CARP). Make it different from VRRP. Field-test the protocol, using your legions of willing lackeys — er, devoted userbase. Make CARP not only a usable replacement for VRRP, but inherently better and stronger. Put the protocol under the BSD license, and give the protocol and code away.

This caused something of a kerfuffle at the time. Ugly accusations flew around. “It’s a VRRP knock-off!” “No, it’s a different protocol!” Great big reams of email were written about the whole thing.

The OpenBSD folks applied to IANA for a protocol number. IANA rejected the application, telling them to use VRRP instead. VRRP was assigned protocol 112. So OpenBSD used protocol 112 for CARP. And putting CARP hosts on a network with Cisco VRRP hosts made Cisco routers crash. The Cisco stack wasn’t robust enough to handle strange packets on the network. Cisco updated their hardware to survive seeing a lone CARP packet.

This escalated the kerfuffle into industry news. You’d see articles in all kinds of industry magazines about OpenBSD versus Cisco.

The OpenBSD folks responded by doing a CARP/VRRP-themed 3.5 release, complete with a Monty Python parody (lyrics, MP3).

And in the end of it all… everyone shut up. Other people started implementing CARP. Because it’s a solid, respectable redundancy protocol. You can get CARP from FreeBSD, Linux, Solaris, and a whole bunch of other vendors…

…including Ciso.

I had plans for today, but I’m too busy laughing. And then I need to go watch some Monty Python.

OpenBSD talk at Farmington Community Library 12 November 2013

I’ll be presenting about OpenBSD at !Michigan/usr/group, a Linux and UNIX user group, on Tuesday, 12 November 2013. The tentative title “OpenBSD for a Linux User Group,” covering the features and culture that make OpenBSD what it is. (Hint: it’s not security.)

These talks are always more fun when readers show up to heckle, throw rotten tomatoes, and question my morals and parentage.

If I have sufficient connectivity and nobody objects, I’ll try to do a Google Hangout for it. But you can’t throw rotten tomatoes over IP. Yet.

FreeBSD Jails and ntpd

I’ve written elsewhere how daemons running on jail servers (the main host, not the imprisoned machines) should listen only on a single address. They shouldn’t bind to all addresses on the machine.

Your average empty FreeBSD install has two problem children: syslogd and ntpd. Adding syslogd_flags="-ss" to /etc/rc.conf handles the first. But FreeBSD’s included ntpd binds to port 123 on all addresses on the machine.

You can run jails while running ntpd. The jail won’t crash in flames. But the jail code expects the jail to have exclusive access to the jail address. This could well come back to bite you later. Besides, it lacks elegance.

Enter openntpd. Openntpd can synch your host clock without binding to any ports. Install it from packages:

# pkg install openntpd

The file /usr/local/etc/ntpd.conf lets you set the preferred server(s) and, if needed, a bind address. This machine is in private address space, so I have to point it at my local time server.

server time.michaelwlucas.com

Now enable openntpd in /etc/rc.conf, and disable the system default ntpd if it’s running.

openntpd_enable="YES"

Run ntpdate to fix the time, then start openntpd.

# ntpdate time.michaelwlucas.com
1 Nov 15:03:22 ntpdate[53689]: adjust time server 192.0.2.130 offset -4.001088 sec
# service openntpd start
Starting openntpd.

The clock is now correct — or, rather, if the clock is wrong, all the servers will be wrong together. And the various jails each has sole access to their own IP addresses.