Consistency in Writing

For the last couple of weeks, the SSH Mastery copyeditor has said “There’s something wrong with Chapter 13, but I can’t figure out what it is.” I told her that I had confidence in her ability to figure it out and to just do her best. (I wasn’t actually confident, but telling her that would have guaranteed that she would not have found it.) The copyedits came back this weekend, along with the following table. Continue reading “Consistency in Writing”

New fiction collection: “Vicious Redemption”

My first collection of short fiction, Vicious Redemption: Five Horror Stories, has started to appear in online bookstores. So far it’s available at Amazon, Barnes & Noble, and Smashwords.

Today it’s ebook only. I have a few marketing things to finalize before it goes to print.

Within a couple weeks, it should appear in other online bookstores. Ebook distribution is faster than physical distribution, but still slower than you’d think. I expect it to be in Kobo & Apple by the end of the month.

Would you enjoy these stories? The first story from the collection, Wednesday’s Seagulls, is posted on my personal web site. Go read it and find out.

I’m giving out review copies to my regular readers. If you normally read this sort of thing, and if you’re willing to read it and post a review on Amazon (as well as anywhere else you’d like), drop me an email.

enable DNSSec resolution on BIND 9.8.1

With BIND 9.8, enabling DNSSec resolution and verification is now so simple and low-impact there’s absolutely no reason to not do it. Ignore the complicated tutorials filling the Internet. DNSSec is very easy on recursive servers.

DNS is the weak link in Internet security. Someone who can forge DNS entries in your server can use that to leverage his way further into your systems. DNSSec (mostly) solves this problem. Deploying DNSSec on your own domains is still fairly complicated, but telling a BIND DNS server to check for the presence of DNSSec is now simple.

In BIND 9.8.1 and newer (included with FreeBSD 9 and available for dang near everything else), add the following entries to your named.conf file.

options {
...
dnssec-enable yes;
dnssec-validation auto;
...
};

This configuration uses the predefined trust anchor for the root zone, which is what most of us should use.

Restart named. You’re done. If a domain is protected with DNSSec, your DNS server will reject forged entries.

To test everything at once, configure your desktop to use your newly DNSSec-aware resolver and browse to http://test.dnssec-or-not.org/. This gives you a simple yes or no answer. Verified DNSSec is indicated in dig(1) output by the presence of the ad (authenticated data) flag.

For the new year, add two lines to your named.conf today. Get all the DNSSec protection you can. Later, I’ll discuss adding DNSSec to authoritative domains.

SSH Mastery Cover Photo

Last summer, preparing for the OpenSSH book, I attended a course on being your own publisher. If you’re interested in publishing, I highly recommend the Think like a Publisher course. The hotel was decorated with a variety of nautical clutter.

This critter hung directly over the breakfast table.

A Real Blowfish
The Hand of Karma

This was obviously the Hand of Fate. I borrowed a couple of really good cameras from fellow workshop attendees and snapped a bunch of photos. I’m a lousy photographer, but with good equipment and enough tries, eventually one came out.

The cover artist has assured me he can strip out the background and arrange this real-life Puffy suitably.

Dec 2011 Updates

The OpenSSH book is in copyedit. I hope to get the copyedits back this year. I’ve seen the first round of copyedits, and they don’t look too bad. Once I make the corrections, the book goes to the print-on-demand layout person and I start on the ebook conversion. The ebook should be out next month.

The best title I’ve had suggested was “SSH: You’re Doing It Wrong.” I love that title, but it’s not really appropriate. Instead, it’ll be “SSH Mastery: OpenSSH, PuTTY, Tunnels, and Keys.” That’s what the book is about, after all.

Progressing on Absolute OpenBSD 2 slowly, thanks to the holidays.

sudo auth via ssh-agent

One of the nicest things about writing a book is that your tech reviewers tell you completely new but cool stuff about your topic. While I was writing the OpenSSH book, one of the more advanced reviewers mentioned that you could use your SSH agent as an authentication source for sudo via pam_ssh_agent_auth.

I have dozens of servers. They all have a central password provider (LDAP). They’re all secured, but I can’t guarantee that a script kiddie cannot crack them. This means I can’t truly trust my trusted servers. I really want to reduce how often I send my password onto a server. But I also need to require additional authentication for superuser activities, so using NOPASSWD in sudoers isn’t a real solution. By passing the sudo authentication back to my SSH agent, I reduce the number of times I must give my password to my hopefully-but-not-100%-certain-secure servers. I can also disable password access to sudo, so that even if someone steals my password, they can’t use it. (Yes, someone could possibly hijack my SSH agent socket, but that requires a level of skill beyond most script kiddies and raises the skill required for APT.)

My sample platform is FreeBSD-9/i386, but this should work on any OS that supports PAM. OpenBSD doesn’t, but other BSDs and most Linuxes do.

pam_ssh_agent_auth is in security/pam_ssh_agent_auth in ports and pkgsrc. There are no build-time configuration knobs and no dependencies, so I used the package.

While that installs, look at your sudoers file. sudo defaults to purging your environment variables, but if you’re going to use your SSH agent with sudo, you must retain $SSH_AUTH_SOCK. I find it’s useful to retain a few other SSH environment variables, for sftp if nothing else.

Newer versions of sudo cache the fact that you’ve recently entered your password, and let you run multiple sudo commands in quick succession without entering your password. This behavior is fine in most environments if you’re actually typing your password, but as sudo will now query a piece of software for your authentication credentials, this behavior is unnecessary. (Also, this caching will drive you totally bonkers when you’re trying to verify and debug your configuration.) Disable this with the timestamp_timeout option.

To permit the SSH environment and set the timestamp timeout, add the following line to sudoers:

Defaults env_keep += "SSH_AUTH_SOCK",timestamp_timeout=0

You can add other environment variables, of course, so this won’t conflict with my earlier post on sftp versus sudo.

Now tell sudo to use the new module, via PAM. Find sudo’s PAM configuration: on FreeBSD, it’s /usr/local/etc/pam.d/sudo. Here’s my sudo PAM configuration:

auth sufficient /usr/local/lib/pam_ssh_agent_auth.so file=~/.ssh/authorized_keys
auth required pam_deny.so
account include system
session required pam_permit.so

By default, sudo uses the system authentication. I removed that. I also removed the password management entry. Instead, I first try to authenticate via pam_ssh_agent_auth.so. If that succeeds, sudo works. If not, the auth attempt fails.

Now try it. Fire up your SSH agent and load your key. SSH to the server with agent forwarding (-A), then ask sudo what you may run.

$ sudo -l
Matching Defaults entries for mwlucas on this host:
env_keep+="SSH_CLIENT SSH_CONNECTION SSH_TTY SSH_AUTH_SOCK",
timestamp_timeout=0

Runas and Command-specific defaults for mwlucas:

User mwlucas may run the following commands on this host:
(ALL) ALL
(ALL) ALL

Now get rid of your SSH agent and try again.

$ unsetenv SSH_AUTH_SOCK
$ sudo -l
Sorry, try again.
Sorry, try again.
Sorry, try again.
sudo: 3 incorrect password attempts

The interesting thing here is that while you’re asked for a password, you never get a chance to enter one. Sudo immediately rejects you three times. Your average script kiddie will have a screaming seizure of frustration.

The downside to this setup is that you cannot use passwords for sudo on the console. You must become root if you’re sitting in front of the machine. I’m sure there’s a way around this, but I’m insufficiently clever to come up with it.

Using the SSH agent for sudo authentication changes your security profile. All of the arguments against using SSH agents are still valid. But if you’ve made the choice to use an SSH agent, why not use it to the fullest? And as this is built on PAM, any program built with PAM can use the SSH agent for authentication.

Moving Static Sites from Apache to nginx

My more complex Web sites run atop WordPress on Apache and MySQL. Every so often, Apache devours all available memory and the server becomes very very slow. I must log in, kill Apache, and restart it. The more moving parts something has, the harder it is to debug. Apache, with all its modules, has a lot of moving parts.

After six months of intermittent debugging, I decided that with the new hardware I would switch Web server software, and settled on nginx. I’d like to switch to Postgres as well, but WordPress’s official release doesn’t yet support Postgres. WordPress seems to be the best of the available evils — er, Web site design tools. The new server runs FreeBSD 9/i386 running on VMWare ESXi. According to the documentations I’ve dug up, it should all Just Work.

Before making this kind of switch, check the nginx module comparison page. Look for the Apache modules you use, and see if they have an nginx equivalent. I know that nginx doesn’t use .htaccess for password protection; I must put my password protection rules directly in the nginx configuration. Also, nginx doesn’t support anything like the mod_security application firewall. I’ll have to find another way to deal with referrer spam, but at least the site will be up more consistently.

To start, I’m moving my static Web sites to the new server. (I’ll cover the WordPress parts in later posts.) I expect to get all of the functionality out of nginx that I have on Apache.

For many years, blackhelicopters.org was my main Web site. It’s now demoted to test status. Here’s the Apache 2.2 configuration for it.

<VirtualHost *:80>
    ServerAdmin webmaster@blackhelicopters.org
    DocumentRoot /usr/local/www/data/bh
    ServerName blackhelicopters.org
    ServerAlias www.blackhelicopters.org
    ErrorDocument 404 /index.html
    ErrorLog "|/usr/local/sbin/rotatelogs /var/log/bh/bh_error_log.%Y-%m-%d-%H_%M_%S 86400 -300"
    CustomLog "|/usr/local/sbin/rotatelogs /var/log/bh/bh_spam_log.%Y-%m-%d-%H_%M_%S 86400 -300" combined env=spam
    CustomLog "|/usr/local/sbin/rotatelogs /var/log/bh/bh_access_log.%Y-%m-%d-%H_%M_%S 86400 -300" combined env=!spam
Alias /awstatclasses "/usr/local/www/awstats/classes/"
Alias /awstatscss "/usr/local/www/awstats/css/"
Alias /awstatsicons "/usr/local/www/awstats/icons/"
ScriptAlias /awstats/ "/usr/local/www/awstats/cgi-bin/"
<Directory "/usr/local/www/awstats/">
    Options None
    AllowOverride AuthConfig
    Order allow,deny
    Allow from all
</Directory>
</VirtualHost>

/usr/local/etc/nginx/nginx.conf is a sparse, C-style hierarchical configuration file. It’s laid out basically like this:

general nginx settings: pid file, user, etc.
http {
    various web-server-wide settings; log formats, include files, etc.
    server {
        virtual server 1 config here
    }
    server {
        virtual server 2 config here
    }
}

The first thing I need to change is the nginx error log. I rotate my web logs daily, and retain them indefinitely, in a file named by date. In Apache, I achieve this with rotatelogs(8), a program shipped with Apache. nginx doesn’t have this functionality; I must rotate my logs with an external script.

In the http section of the configuration file, I tell nginx where to put the main server logs.

http {
...
error_log /var/log/nginx/nginx-error.log;
access_log /var/log/nginx/nginx-access.log;

Define a virtual server and include the log statements:

http {
...
    server {
        server_name blackhelicopters.org www.blackhelicopters.org;
        access_log /var/log/bh/bh-access.log;
        error_log /var/log/bh/bh-error.log;
        root      /var/www/bh/;
    }
}

That brings up the basic site and its logs. I don’t need to worry about the referral spam log, as I cannot separate it out. nginx doesn’t need ServerAlias entries; just list multiple server names.

To test the basic site, make an /etc/hosts entry on your desktop pointing the site to the new IP address, like so:

139.171.202.40 www.blackhelicopters.org

You desktop Web browser should use /etc/hosts over the DNS entry for that host, letting you call up the test site in your Web browser. Verify the site comes up, and that nginx is actually serving your content. Verify that the site’s access log contains your hits.

To rotate these logs regularly, create a script /usr/local/scripts/nginx-logrotate.sh.

#!/bin/sh

DATE=`date +%Y%m%d`

#main server
mv /var/log/nginx/nginx-error.log /var/log/nginx/nginx-error_$DATE.log
mv /var/log/nginx/nginx-access.log /var/log/nginx/nginx-access_$DATE.log

#bh.org
mv /var/log/bh/bh-error.log /var/log/bh/bh-error_$DATE.log
mv /var/log/bh/bh-access.log /var/log/bh/bh-access_$DATE.log

killall -s USR1 nginx

Run at 11:59 each night via cron(8).

59 23 * * * /usr/local/scripts/nginx-logrotate.sh

This won’t behave exactly like Apache’s logrotate. The current log file won’t have the date in its name. There will probably be some traffic between 11:59 PM and the start of the new day at 12:00AM. But it’s close enough for my purposes.

I must add entries for every site whose logs I want to rotate.

Now there’s the aliases. I don’t have awstats running on this new machine yet, but I want the Web server set up to support these aliases for later. Besides, you probably have aliases of your own you’d like to put in place. Define an alias within nginx.conf like so:

location ^~/awstatsclasses {
    alias /usr/local/www/awstats/classes/;
}
location ^~/awstatscss {
    alias /usr/local/www/awstats/css/;
}
location ^~/awstatsicons {
    alias /usr/local/www/awstats/icons/;
}

Finally, I need my home directory’s public_html available as http://www.blackhelicopters.org/~mwlucas/. This doesn’t update, but people link here. The following snippet uses nginx’s regex functionality to simulate Apache’s mod_userdir.

location ~ ^/~(.+?)(/.*)?$ {
    alias /home/$1/public_html$2;
    index  index.html index.htm;
    autoindex on;
}

For most sites, I would define a useful error page. The purpose of this site is to say “don’t look here any more, look at the new Web site,” so pointing 404s to the index page is reasonable. Defining an error page like so:

error_page 404 /index.html;

The configuration for this entire site accumulates to:

server {
    server_name blackhelicopters.org www.blackhelicopters.org;
    access_log /var/log/bh/bh-access.log;
    error_log /var/log/bh/bh-error.log;
    root      /var/www/bh/;
    error_page 404 /index.html;
    location ^~/awstatsclasses {
        alias /usr/local/www/awstats/classes/;
    }
    location ^~/awstatscss {
        alias /usr/local/www/awstats/css/;
    }
    location ^~/awstatsicons {
        alias /usr/local/www/awstats/icons/;
    }
    location ~ ^/~(.+?)(/.*)?$ {
        alias /home/$1/public_html$2;
        index  index.html index.htm;
        autoindex on;
    }
}

While I’m happy with nginx performance so far, I’m only running a couple of static sites on it. The real test will start once I use dynamic content.

Why I Give Books Away

For a year or so I’ve wanted to write a post about the impact of book reviews, specifically on Amazon book reviews, but Anne R. Allen has saved me the trouble.

In short: Amazon owns my writing career.

They make their decisions based on reviews by people like you.

And when I say “people like you,” I mean you, personally.

The biggest thing you can do to help any author is review their book in twenty words or more, and rate it four or five stars, and post it on Amazon. (Amazon considers a 3-star review not average, but negative.)

Today, for good or ill, Amazon owns the book business. Especially tech authors. We live and die by Amazon reviews. Reviews on other sites are nice too, but if the review isn’t on Amazon, it’s mostly shouting into the echo chamber.

mirroring FreeBSD-9 disks with GPT

I recently tried to mirror my hard drives in a new machine. The Handbook instructions, and those in my own Absolute FreeBSD, didn’t work well. (The Handbook now warns about this in a big, friendly, hard-to-miss red box.) So how can I mirror my disk? By using per-partition mirroring rather than full-disk mirroring.

I should note up front that this article is the result of my researches and testing. I am not a filesystem developer. I’m not even a FreeBSD committer any more. You should check the FreeBSD Handbook for updated documentation before trying this approach.

First, I need to partition my disks identically. My system has two disks, da0 and da1. da0 has an installed system, da1 is blank. Use gpart(8) to copy the GPT.

# gpart backup da0 > da0.gpt

The file should look something like this:

# cat da0.gpt
GPT 128
1 freebsd-boot 34 128
2 freebsd-ufs 162 201326464
3 freebsd-swap 201326626 8388540

Now copy this to the second disk.

# gpart restore -F /dev/da1 < da0.gpt

The -F flag tells gpart to destroy any existing GPT on the target disk. Now verify the GPT on both disks.

# gpart show
=> 34 209715133 da0 GPT (100G)
34 128 1 freebsd-boot (64k)
162 201326464 2 freebsd-ufs (96G)
201326626 8388540 3 freebsd-swap (4G)
209715166 1 - free - (512B)

=> 34 209715133 da1 GPT (100G)
34 128 1 freebsd-boot (64k)
162 201326464 2 freebsd-ufs (96G)
201326626 8388540 3 freebsd-swap (4G)
209715166 1 - free - (512B)

These look pretty identical to me. I now have a separate device node for each partition on each disk. Mirroring these works much like mirroring an entire disk.

Unlike mirroring MBR disks, to mirror GPT partitions you must be in single-user mode. Now that GPT is the default partitioning scheme I’m sure someone will figure out a clever way around this, but for now reboot into single-user mode.

# gmirror label -vb round-robin p1 /dev/da0p1
# gmirror label -vb round-robin p2 /dev/da0p2
# gmirror label -vb round-robin p3 /dev/da0p3

/dev/da0p3 is the default swap partition. You must decide if you want to mirror your swap partitions as well. I’m choosing to do so. If one of my disks fails, the system has a fighting chance to continue running. Having two independent swap areas, one on each disk, means that a disk failure will yank a swap space out from under the otherwise-working system.

Now add the second disk’s partitions to your mirror devices.

# gmirror insert p1 /dev/da1p1
# gmirror insert p2 /dev/da1p2
# gmirror insert p3 /dev/da1p3

This will mirror your boot blocks, your root disk, and your swap space.

Now you must update /etc/fstab to boot from your mirror. The tricky bit here is that the system now has /dev/da0p2 mounted as root, read-only. You don’t want to write to /dev/da0p2 again; all writes should go to the mirror. Instead, mount /dev/mirror/p2 to a temporary location.

# mount /dev/mirror/p2 /mnt
# cd /mnt/etc
# cp fstab fstab-old
# ee fstab

vi requires a read-write /var/tmp, but ee works just fine.

Let the system run until the disks are synchronized. Check your disk status with “gmirror status.”

# gmirror status
Name Status Components
mirror/p1 COMPLETE da0p1 (ACTIVE)
da1p1 (ACTIVE)
mirror/p2 DEGRADED da0p2 (ACTIVE)
da1p2 (SYNCHRONIZING, 17%)
mirror/p3 DEGRADED da0p3 (ACTIVE)
da1p3 (SYNCHRONIZING, 7%)

After your disks synchronize, reboot. Don’t just exit single-user mode, as you have the wrong root partition mounted. (I found that rebooting before mirror synchronization meant the system came up with a read-only /.)

Your drives are now mirrored. Don’t forget to add mirror checks to your daily status mails. Add the following to /etc/periodic.conf:

daily_status_gmirror_enable="YES"

Hopefully, we’ll have a faster way to do this soon.

UPDATE 7/12/2011: You must have geom_mirror_load=YES in /boot/loader.conf to run gmirror commands.