my OpenSSH AuthorizedKeysCommand script

The bleeding-edge OpenSSH supports using an AuthorizedKeysCommand statement in sshd_config to get the authorized_keys file for a user. This lets you store your authorized_keys files in LDAP, but avoids linking OpenSSH against OpenLDAP. (You could actually use any data store for your back end, but LDAP is both the most popular and what I have.)

Your AuthorizedKeysCommands script should take one argument, and return a series of authorized keys, one per line. CentOS has a script, which I previously mentioned, and one of my readers was kind enough to put together an OpenBSD port for it.

But there’s some problems with the CentOS approach, from a cross-platform perspective. While it’s fine on CentOS, porting and installing it on other platforms is more difficult than it needs to be. I’d really like something that has very few dependencies, is easy to install, and is easily portable across platforms.

When setting up a couple of Ubuntu boxes I came across an Ubuntu AuthorizedKeysCommand script. His approach is much simpler than the CentOS approach, but his script didn’t work for me out of the box, and it wouldn’t work on other operating systems without modification. But it inspired me to write my own script, one that works across all of my operating systems, using only the tools I already have on all of my servers. My results follow. But before I offer my script, I feel obliged to warn you.

The bad news is, I am not a programmer. I have been told by multiple independent parties that my code makes the Baby Jesus cry.

The really bad with this is that you need an ldapsearch command that returns your public keys, and this command is hard-coded with this script. The ideal script would read the system ldap config, but you’d probably have to hard-code a path to ldap.conf anyway.

The appalling news is, I wrote it in Perl. Why? Well, the evidence indicates that I am insane. But every server in my environment runs Perl, and it’s been ten years since I used enough sed/awk to make this work. A proper authentication credentials script would not use Perl.

I’m not suggesting that you take my code. I’m hoping to motivate one of my programmer readers, whose code only makes the Baby Jesus grimace a little, to do better. I mean, I’ve set a pretty low bar here.


#!/usr/bin/perl
#takes one argument, the username to get the keys for
die unless $ARGV[0];

open (LDAP, "/usr/bin/ldapsearch -L -xZb\"dc=michaelwlucas,dc=com\" '(&(objectClass=posixAccount)(uid=$ARGV[0]))' sshPublicKey |") || die "ldapsearch failed $!\n";

while (<LDAP>) {
next if /^#|^version|^dn\:|^\s*$/;
s/\n//;
s/\://g;
s/sshPublicKey/\n/;

s/^ //;
print;
}
print "\n";

One catch with this is that you’re processing the raw output of ldapsearch. An acceptable ldapsearch will return keys that look something like this:


...
sshPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAw9zmtbk8bT+7OVpVvzuYYQqRhF8j
...blah, blah, blah...
ilXzBw== rsa-key-20110114

Initially, my LDAP searches returned some keys that looked like this:
...
sshPublicKey:: c3NoLWRzcyANCkFBQUFCM056YUMxa2MzTUFBQUNCQVBhZmNhemdmUHI1T29DdFp
...blah, blah, blah...
y5sb2NhbA==

This key doesn’t start with ssh-rsa or ssh-dss, and it doesn’t even look like a SSH key. But if you look at the key in a graphical tool like phpLDAPAdmin, it has the leading ssh-rsa. In my case, the key had an extra carriage return at the end of the key, which meant that ldapsearch had to base64 encode it. Get rid of extra whitespace. The LPK patch doesn’t notice that space, but a simple external script does.

Of course, if someone wanted to write an AuthorizedKeysCommand that was portable, handled that encoding, and didn’t suck as bad as mine, it wouldn’t hurt my feelings at all. Really.

Conference Submissions & BSDCan 2013

The BSDCan 2013 call for proposals is open. As a BSDCon committee member, it’s my responsibility to get you lot to submit interesting papers. So: submit!

More than once, I’ve been asked something like “How can I get a paper accepted at BSDCan?” or “Why was my BSDCan paper rejected?” Here’s my answer to that general question. Lots of this applies to any conference, but as I’m not on the committees for those other conferences, I can’t claim any authority there. Other conferences have their own ideas, and other BSDCan committee members might have independent thoughts as well.

The purpose of a BSDCan paper is not to advertise the presenter’s work, but to put butts in seats. Sometimes these overlap. I want someone looking at the BSDCan schedule to say “Wow, I HAVE to go to this!” But “be interesting” isn’t exactly useful advice. Some of this might be.

1) Present an topic that hasn’t been covered before. The previous proceedings are all available online. Some topics come up every year. Don’t submit those unless you have a fresh new take on them.

2) Tell us why YOU should be the one to present your topic. We frequently get multiple similar proposals, and have to choose between them based on who we think might do the best job. Don’t assume that everyone on the committee knows who you are. One of us probably knows that you run weaselBSD on twelve thousand workstations, but they probably won’t share than information with everyone else. (Paper selection is done via web form, which isn’t as conducive to back-and-forth discussion as in-person or mailing list discussions.) More than once I’ve had someone come up to me at a conference and say “Why did you have so-and-so report on X? I have forty times as much X as the speaker you chose!” My answer is, “You didn’t TELL US that you had so much.” I am not telepathic.

3) We get many FreeBSD proposals. I want more Net/Open/Dfly proposals, as well as for projects and devices based on them. Non-F/N/O/D BSD papers have much smaller audiences, so a paper on them has to be very interesting. Again, butts in seats.

4) Include “lessons learned.” Anybody can read a man page. If you were involved in recovering from the recent FreeBSD security incident, you should submit a paper about it. If you deployed OpenBSD in a nuclear submarine and learned about the importance of the little radiation-detection badge, you should submit a paper. “Lessons learned” implies that you made mistakes, and can upgrade your talk from the merely technical to a bit of a story. Don’t be afraid to embarrass yourself, we’ve all been there. Sharing mistakes helps connect your presentation to the audience.

5) Tell us what the audience will get out of the talk — e.g., if it’s a sysadmin talk: “Attendees will be able to configure weaselBSD’s anti-ferret features and understand how they integrate with email and Web.”

6) Spelling and grammar are not pimple cream, to be dabbed on where needed. If you’re going to indulge in floccinaucinihilipilification, you’d better be able to spell it.

7) Speaking of floccinaucinihilipilification, you’re better off saying why your stuff is awesome as opposed to running down someone else’s work. I know many reasons why computers suck. Learning something to make my job suck less is good.

That’s what I look for, at least. Others have their own opinions. And if you think a little bit, you can make your proposal answer all of these. Unless you’re running weaselBSD, that is…

Now: submit!

OpenBSD ruleset tracing

As Henning reviews the Absolute OpenBSD manuscript, he’s pointed out items that I’ve missed. Some of these are only documented in man pages, while others don’t seem to really be documented anywhere except in the source code. Here’s an interesting tidbit he pointed out that I haven’t seen anywhere other than Henning’s email. (Having said this in public, I’ll now find all sorts of examples that I missed, such as Henning’s slides from EuroBSDcon 2010.)

You can log specific connections to separate log devices, for simpler debugging. You need the log rule near the beginning of your ruleset, but it doesn’t have an effect on whether a packet is passed or blocked, or how it’s translated. Here’s a pf.conf line to tell PF to watch for connections from 192.0.2.226 to 203.0.113.34.

match log (matches) from 192.0.2.226 to 203.0.113.34

Now watch the log interface, searching for the destination address:

# tcpdump -n -e -ttt -i pflog0 ip host 203.0.113.34
tcpdump: WARNING: snaplen raised from 116 to 160
tcpdump: listening on pflog0, link-type PFLOG
Dec 17 17:40:25.071592 rule 2/(match) block out on fxp0: 192.0.2.226.14033 > 203.0.113.34.822: S 2830367545:2830367545(0) win 16384 (DF)
Dec 17 17:40:25.071600 rule 5/(match) pass out on fxp0: 192.0.2.226.14033 > 203.0.113.34.822: S 2830367545:2830367545(0) win 16384 (DF)
Dec 17 17:40:25.071604 rule 5/(match) pass out on fxp0: 192.0.2.226.14033 > 203.0.113.34.822: S 2830367545:2830367545(0) win 16384 (DF)
^C

You get the number of every rule that matches the packet. If you use NAT, it prints original and destination addresses. With my wussy PF rules, this isn’t terribly exciting. With more complicated rules, however, it makes debugging rulesets much easier.

This is the sort of useful information you’ll get if you preorder the new Absolute OpenBSD. (See how I subtly slipped that in? I’m getting good at this marketing stuff.)

Absolute OpenBSD pre-orders now available

No Starch Press now has pre-orders for new Absolute OpenBSD. Order direct from the publisher, and get both the ebook and the paper for one price. If you use the coupon code ILUVMICHAEL you’ll get a discount, and I get a commission on the sale. (Bolded 20130207 because more than one person has said they missed that line.) If you use another coupon code, I still get paid, but not as much. I’m not deeply concerned which way you buy it, so long as you buy it.

Here’s the cover of the new edition. It incorporates art from the first edition, plus a new background.AO2e Cover

On a vaguely related note, I recently saw a link to my blog from a Chinese Unix users message board. Curious, I asked Google Translate what it said. It’s a discussion of the new book, which is awesome. Slightly worrying, though, is that in the translation they repeatedly refer to me as “Great God Lucas” or some variant thereof. I’m hoping that this is an artifact of translation, or some cultural thing I was previously unaware of.

Otherwise, it would seem that I have a cult of worshippers in China, and that I must learn Chinese in order to issue my commands.

Even more tangentially, links within a translated page take you to a translated version of that page. That’s pretty cool.

1st draft of Absolute OpenBSD, 2nd Ed. complete

Last night, I finished the first draft of the new edition of Absolute OpenBSD.

This is the longest book I’ve ever written (23 chapters). It’s taken longer than any other nonfiction book (3 years). Now that a first draft exists, I can state with some confidence that the book will be out about next spring-ish.

As a first draft exists, if I get trampled by a rabid caribou between now and then, the book will still come out.

This weekend is the first time in years that I will have had no work to do on the book. (Unless Henning sends me corrections on the few chapters he has left.) I plan to gaze blankly into space for several hours.

Absolute OpenBSD 2nd Edition status, 15 November 2012

Chapters 1-22 are written. Only chapter 23 remains.

The first 23 chapters are either in preliminary tech review (Henning Brauer), editing (No Starch Press), technical review (Peter Hansteen), or copyediting (No Starch Press). And every time any one of those folks are done, the chapter comes back to me for rewrites. Which is as it should be, of course… unlike some publishers, NSP gives me every chance to improve the book, as opposed to having some unpaid intern with a degree in medieval lit “fix” the text.

One chapter to go. Back to writing…

Easy Security Project: standalone ssh-ldap-helper

I’ve been waiting for quite a while for an official way to centrally manage user authentication keys in OpenSSH. If you have a dozen servers, copying authorized_keys files around is a pain. If you have more than that, it’s really really painful. The OpenSSH guys have had good reasons for not wanting to link LDAP libraries straight into OpenSSH. They also gave some general guidance of what they’d want to see in a patch that supported LDAP authentication.

Jan Chadima from Redhat took OpenSSH up on this, wrote a patch as per spec, and submitted it to OpenSSH. And Damien Miller committed it. LDAP support for OpenSSH will be in 6.2…

…sort of.

The patch adds support for getting a user’s authorized_keys file from a helper program. Redhat includes a helper program, ssh-ldap-helper. That program is not in the OpenSSH patch. And, truthfully, there’s no reason it should be in the main OpenSSH distribution. We’ll see helpers for LDAP, for database lookups, for FUSE and HTTP and whatever weird data storage people come up with. I don’t want the OpenSSH guys spending their time writing these helpers.

But the source code for ssh-ldap-helper is in the Red Hat source RPM. As far as I can tell, it’s under a BSD license.

If you’re looking for a way to contribute to the OpenSSH user community, however, digging into the RPM (it’s just a tarfile), extracting the included OpenSSH code, and adding the patch for ssh-ldap-helper, ssh-ldap-wrapper, and the man page is pretty easy. I got that far, after all! I imagine that someone with a little bit of knowledge could make it compile on xBSD. Or at least, it’s a place to start.

You’d make my life a lot easier. And give me more time to finish the new edition of Absolute OpenBSD. That’s what you lot want me to do with my time, isn’t it? (I’ll have a post on that status in a few days.)

I also have to give props to Red Hat on this. They had a need in OpenSSH. They were given the requirements for that need to be met in mainline OpenSSH. And they met those needs and submitted the patch. Everyone cooperated, everyone gets what they need. That is how open source should work. Given how some other open source companies and projects are behaving lately, this makes me feel pretty good about the BSD community.

Get Your Haiku Published in the new “Absolute OpenBSD”

Something weird happened as I worked on the second edition of Absolute OpenBSD: people started sending me haiku. The first edition included a haiku at the beginning of each chapter, something apropos to the topic.

TCP/IP
Learn how it fits together
You cannot escape

I reviewed the old book before outlining the new version, and the haiku made me wince. They’re mediocre at best. I considered dropping them from the new edition, or perhaps replacing them with quotes on trust, but an informal Twitter poll came out overwhelmingly in favor of the haiku. This demonstrates that computing professionals have lousy taste in poetry, or that an author is permitted no opinion on the quality of his own work. Or both.

Frankly, the haiku my fans send are better than the ones I write. Some of mine are okay, but they can’t compete with someone else’s inspiration.

So, here’s the deal:

You’ll find the outline for the second edition in my September status blog post. Each chapter needs a haiku.

Post your English-language haiku here, along with valid contact information and your name as you’d like to be credited. If your haiku is better than what I have for that chapter, I’ll use yours instead of mine. By posting your haiku here, you give me permission to use it in the book. Winners will be selected by me, at my sole discretion, based on whatever criteria I feel like using at the time. Your best bet is to amuse me.

If you don’t want to post your haiku, you can email it to me. Use the subject of “ao2e haiku” to avoid the Horrible Black Void that awaits most email I receive.

What is a haiku? Real haiku are in Japanese. I can’t use real haiku — I can’t even read real haiku. For my purposes, a haiku has:

  • 5-syllable first line, 7-syllable second line, 5-syllable third line
  • A season word (i.e., summer, snow, etc)
  • A comparison
  • You might note that my leading haiku breaks two of these three rules. It amuses me, however, which is more important than any other characteristic. But if you can follow all three rules in a haiku about packet filtering, I’ll be slightly impressed.

    Both entries and attributions must be PG-rated. As in, no obscenity. Sorry, folks, I know that obscenity is a staple in sysadmin circles, but AO2e is supposed to be a clean family book.

    I’m not limiting entries per person, but I can say that if you flood me with dozens of mediocre haiku I’ll probably miss the the one awesome one you do post. (“Oh, it’s him again. Sigh.”)

    So, what’s in it for you?

    Selected haiku will appear at chapter headings in the second edition of Absolute OpenBSD, with attribution. This is your chance at eternal fame. Selected haiku-ists will get an ebook of the finished book. If I can swing a sufficient number of physical copies, I’ll give those out as well. Depends on how many winners and how many copies I get.

    Competition will remain open until I finish the first draft of the book. I’m writing frantically, hoping to get a first draft done by mid-November. If I make that deadline, the book can exist for BSDCan 2013. That would be awesome. Can I make that deadline? Dunno. I’m holding the contradictory ideas “no, that’s impossible” and “sure I can!” in my brain simultaneously.

    So, in closing:

    Lucas is lazy
    Your haiku makes him chortle?
    Get free electrons.

    Log Only sudo Failures

    The sudo(8) privilege management tool is very admin-friendly in that it logs successes and failures. I don’t really care when my users successfully use sudo. I do care when they use it unsuccessfully, however. A sudo failure indicates that either the user doesn’t know their system password, or they’re trying to use forbidden commands.

    sudo keeps logs. The interesting thing is, successful log messages are of priority notice, while unsuccessful attempts are of priority alert. This opens up an easy way to improve security and customer service.

    First, a user who cleverly gets root can edit your log files. forward your sudo logs to your logging host. Your users should not have access to your logging host.

    First, split your sudo logs out into two logs. You can set sudo’s syslog facility, but as I’m always short on facilities, I tend to break sudo out via program name. Here’s a syslog.conf entry.

    !sudo
    *.* /var/log/sudo
    *.alert /var/log/sudofail

    Touch the files, restart syslogd, and /var/log/sudofail will contain only password failures and attempts to run forbidden commands. The two log entries are very different.

    Sep 26 10:37:27 caddis sudo: mwlucas : command not allowed ; TTY=ttyp0 ; PWD=/home/mwlucas ; USER=root ; COMMAND=/sbin/reboot
    Sep 26 10:53:09 caddis sudo: mwlucas : 3 incorrect password attempts ; TTY=ttyp0 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/su

    Separating this log out opens some interesting customer service possibilities. If you’re on OpenBSD, you can automatically have newsyslog email you the log of failures. Otherwise, you can set up a separate script to do that, or feed it to your alerting system, or whatever. Then have a helpdesk minion call the user in question and ask what they’re trying to do. Perhaps they’ve forgotten their password. Perhaps someone else got access to their account. Perhaps they’re having trouble. Maybe they need sudo -l explained to them.

    The end user will either feel like you’re watching out for them, or realize that your sysadmin group watches the systems very closely.

    Even if you don’t take proactive action, having sudo failures logged to a separate file simplifies digging through the logs.

    OpenBSD read-only ports tree with restrictive sudo

    The OpenBSD folks strongly encourage users to use packages for software management. Most of the time, their packages just work. But sometimes, you must use a port.

    OpenBSD includes an updated Apache 1.3 server, and recommends that everyone use it if at all possible. (There’s also nginx, which is the future platform, but it’s not quite integrated yet.) I have a Web application that only runs on Apache 2.2, so the included Web server is not an option. OpenBSD provides an Apache 2.2 package for people like me, which is very kind of them. But I need an Apache 2.2 with LDAP authentication support. That means I must build Apache 2.2 from a port.

    If I have to use ports, then I want to do so as easily as possible. When I need to upgrade my ports, I want to be able to remove /usr/ports and extract the tarball that goes with whatever snapshot I’m running. I need the ports tree to do all its work, and store all its packages, outside the ports tree itself. This means a read-only ports tree.

    I’m running the August 22 i386 snapshot everywhere. I build packages from ports on one machine and share out the package repo via NFS.

    I dislike running as root for routine tasks, like building ports on the port-building machine. The ports tree supports using sudo for privileged operations. I don’t want to be continually interrupted to enter my password, though. And I don’t want to give unlimited root access via sudo without a password. This means that I need to lock down my account on this machine to only those activities needed to build packages from ports. I readily concede that building packages requires high-level privileges, but there’s a world of difference between rm -f /usr/ports/* and rm -rf /*. Could an intruder exploit this? Absolutely. You must run make(1) as root to build a port, and you can run fdisk(8) via make. But it will protect me from operator error. And my operators make errors.

    I also want my minions to be able to build packages without giving out root. Because, you know, logging into a system and typing a command because someone else needs a package is extra work for me.

    So, how to do this?

    First, create a system group for people who may build packages. This group contains two users, myself and lasnyder. From /etc/group:

    portbuild:*:10001:mwlucas,lasnyder

    My /home partition has lots of space, so I’ll build everything there. First, we need four directories:

  • one for building stuff: /home/ports/wrkobjdir
  • one for completed packages: /home/ports/pkgrepo
  • distfiles: /home/ports/distdir
  • package plist database: /home/ports/plist

    Create these directories, and set their ownership (as well as /home/ports) for group writing.

    # chgrp portbuild /home/ports/*
    # chmod 775 /home/ports/*

    Any user in the portbuild group can write to these directories.

    Now tell the ports system about these directories. Make the following entries in /etc/mk.conf:

    WRKOBJDIR=/home/ports/wrkobjdir
    DISTDIR=/home/ports/distdir
    PACKAGE_REPOSITORY=/home/ports/pkgrepo
    PLIST_DB=/home/ports/plist
    SUDO=/usr/bin/sudo

    (The sudo isn’t necessary for the directories, but I’m not going to send you back later to add it. That would be lame.)

    Now for sudo. Give everyone in the portbuild group permission to run any command in the PORTBUILDCMDS alias.

    %portbuild ALL= NOPASSWD: PORTBUILDCMDS

    Now create the PORTBUILDCMDS alias. I built this alias iteratively: build a port, wait for the build to fail, add the missing command with the tightest restrictions that seem sensible, clean the port, and remake it. The following alias was sufficient for everything I tried:

    Cmnd_Alias PORTBUILDCMDS = /usr/bin/install, /usr/sbin/chown, /bin/chgrp, /bin/sh -c umask, /usr/sbin/mtree, /usr/bin/touch, /usr/bin/env, /usr/sbin/pkg_create, /bin/rm -f /home/ports/pkgrepo/*, /usr/bin/make, /usr/bin/perl /usr/ports/infrastructure/bin/*, /bin/chmod 555 /home/ports/*, /bin/mkdir -p /home/ports/*, /bin/rm -rf /home/ports/*

    Now choose a port and build it.

    # cd /usr/ports/editors/vim
    # make clean && make

    (When testing, I always clean a port before building it.)

    You might find that the build stops and you’re asked for a password. This means that sudo is trying to run a command that’s not in your command alias. Go ahead and enter your password. The build will fail, because you don’t have privileges, but you’ll get an error message in /var/log/secure. Between the error in the terminal window and the error in the log file, you should be able to figure out exactly which command failed.

    It’s impossible to know ahead of time every command that will ever be used by any port that ever exists. This iterative process is a pain at first, but once you’ve built a few ports you’ll find most of the necessary commands. The sudoers command alias I include here was sufficient to build editors/vim, which calls in python, dbus, glib, three different autoconfs, tcl/tk, CUPS, and a whole bunch of other crap. (I don’t use vim myself, mind you, but if you want a port that hauls in whole bunches of stuff, it’s a good choice. I could have built Emacs, but I wanted the build to finish today.)

    In building the first port, the ports system creates a temp directory, /tmp/portlocks. The ports system doesn’t use sudo to access this directory, and the directory is owned by the user who built the first port on this system. Change the group and assign group privileges to this directory.

    # chgrp portbuild /tmp/portslocks/
    # chmod 775 /tmp/portslocks/

    (Is this a bug, or a feature. I dunno. But I’m sure that some reader will tell me.)

    It seems that not all ports can be built without running as root. This isn’t a usual configuration, so I’m not shocked that not all code paths are tested — especially when building random software from random authors. When I tried to build devel/autoconf/2.59, I got:

    ===> Building package for autoconf-2.59p3
    Create /home/ports/pkgrepo/i386/all/autoconf-2.59p3.tgz
    Warning: @option no-default-conflict without @conflict
    mv: rename /home/ports/pkgrepo/i386/tmp/autoconf-2.59p3.tgz to /home/ports/pkgre po/i386/all/autoconf-2.59p3.tgz: Permission denied
    *** Error code 1

    I reported the error to ports@ like a good little user. It’s a holiday weekend, so I’m also not surprised I haven’t heard back.

    I only hit this error after building fifty-odd ports, though. It appears that limited sudo permissions are doable.