Next Nonfiction Book

I’ve made it a practice to not announce book topics or titles until the book is well underway. Writing a big book takes not less than a year (Absolute FreeBSD) and up to three years (Absolute OpenBSD, 2nd ed). Once I hand in the completed first draft to the publisher, there’s editing, tech edits, copyedit, page layout, and so on. It’s a few months to get the book into production.

Delaying the announcement also gives me the chance to determine if the book is realistic. I’ve made no secret that I write about topics that I’m not qualified to cover. I’ve had more than one tech book that I’ve started, only to discover three chapters in that I am so not the person to write this book. Delaying announcing the topic gives me a chance to back out without anybody knowing.

I’m trying something a little different this time. My next book will be published by Tilted Windmill Press (my private label) and much smaller than my BSD tomes. I have an outline. I’ve done the reading. My educational lab work is done (meaning that my rate of screaming “Why isn’t this working?” has dropped from thrice hourly to twice daily). And I’m doing a fairly wide variety of work with the topic in the next six months.

The next book is on (drum roll please): DNSSec. Blame Richard Bejtlich. (I wish I could find the tweet in question, but seriously, how am I supposed to resist him declaring “You’re our only hope?” Flattery will get you anywhere. Especially if you’ve given me enough cover quote copy to last the rest of my career.)

Writing the book concurrently with implementing DNSSec across great big piles of domains with multiple registrars should give me all sorts of problems to write about, and give my readers more benefit from my real-world pain.

I know a lot of people don’t like DNSSec, have cogent arguments why DNSSec is poo, and really wish it would go away. They take me writing a book about it as a refutation of their arguments. It’s not. But DNSSec is here. It’s the standard. We’ve got to deal with it. And the supporting software has improved to the point where DNSSec can be implemented by the typical overworked sysadmin, rather than only crypto fans.

DNSSec also gets you things like SSHFP records and vendor-free SSL certificates. The former is convenient. The latter will eliminate any excuse for unencrypted communications.

Why announce this ahead of time? For one, you’ll probably see me griping about random pieces of DNSSec boneheadedness on Twitter. The savvy will be able to guess. Announcing the book will help keep my nonfiction writing focused. It’s still possible that someone will rush a book into print ahead of me, but the shorter cycle of independent publishing reduces that risk. The audience and community reaction to SSH Mastery is also encouraging; I know that if I write a good book, my readers will tell others about it, regardless of the publisher. If someone beats me to print, my readers will still support me.

And if I write a crap book, it deserves to fail.

(As an aside: having readers who tell their friends and co-workers about my books is freaking awesome. I could not publish books if you didn’t support my work. Thank you.)

Ideally, I’ll have this book out for BSDCan 2013. Tilted Windmill Press is the BSDCan T-shirt sponsor, so having a book out for the conference would be a good idea.

More questions? Too bad. That’s all I know right now. Except that now that I’ve set and announced a goal, my life will go horribly askew specifically to delay me.

Easy Security Project: standalone ssh-ldap-helper

I’ve been waiting for quite a while for an official way to centrally manage user authentication keys in OpenSSH. If you have a dozen servers, copying authorized_keys files around is a pain. If you have more than that, it’s really really painful. The OpenSSH guys have had good reasons for not wanting to link LDAP libraries straight into OpenSSH. They also gave some general guidance of what they’d want to see in a patch that supported LDAP authentication.

Jan Chadima from Redhat took OpenSSH up on this, wrote a patch as per spec, and submitted it to OpenSSH. And Damien Miller committed it. LDAP support for OpenSSH will be in 6.2…

…sort of.

The patch adds support for getting a user’s authorized_keys file from a helper program. Redhat includes a helper program, ssh-ldap-helper. That program is not in the OpenSSH patch. And, truthfully, there’s no reason it should be in the main OpenSSH distribution. We’ll see helpers for LDAP, for database lookups, for FUSE and HTTP and whatever weird data storage people come up with. I don’t want the OpenSSH guys spending their time writing these helpers.

But the source code for ssh-ldap-helper is in the Red Hat source RPM. As far as I can tell, it’s under a BSD license.

If you’re looking for a way to contribute to the OpenSSH user community, however, digging into the RPM (it’s just a tarfile), extracting the included OpenSSH code, and adding the patch for ssh-ldap-helper, ssh-ldap-wrapper, and the man page is pretty easy. I got that far, after all! I imagine that someone with a little bit of knowledge could make it compile on xBSD. Or at least, it’s a place to start.

You’d make my life a lot easier. And give me more time to finish the new edition of Absolute OpenBSD. That’s what you lot want me to do with my time, isn’t it? (I’ll have a post on that status in a few days.)

I also have to give props to Red Hat on this. They had a need in OpenSSH. They were given the requirements for that need to be met in mainline OpenSSH. And they met those needs and submitted the patch. Everyone cooperated, everyone gets what they need. That is how open source should work. Given how some other open source companies and projects are behaving lately, this makes me feel pretty good about the BSD community.

Log Only sudo Failures

The sudo(8) privilege management tool is very admin-friendly in that it logs successes and failures. I don’t really care when my users successfully use sudo. I do care when they use it unsuccessfully, however. A sudo failure indicates that either the user doesn’t know their system password, or they’re trying to use forbidden commands.

sudo keeps logs. The interesting thing is, successful log messages are of priority notice, while unsuccessful attempts are of priority alert. This opens up an easy way to improve security and customer service.

First, a user who cleverly gets root can edit your log files. forward your sudo logs to your logging host. Your users should not have access to your logging host.

First, split your sudo logs out into two logs. You can set sudo’s syslog facility, but as I’m always short on facilities, I tend to break sudo out via program name. Here’s a syslog.conf entry.

!sudo
*.* /var/log/sudo
*.alert /var/log/sudofail

Touch the files, restart syslogd, and /var/log/sudofail will contain only password failures and attempts to run forbidden commands. The two log entries are very different.

Sep 26 10:37:27 caddis sudo: mwlucas : command not allowed ; TTY=ttyp0 ; PWD=/home/mwlucas ; USER=root ; COMMAND=/sbin/reboot
Sep 26 10:53:09 caddis sudo: mwlucas : 3 incorrect password attempts ; TTY=ttyp0 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/su

Separating this log out opens some interesting customer service possibilities. If you’re on OpenBSD, you can automatically have newsyslog email you the log of failures. Otherwise, you can set up a separate script to do that, or feed it to your alerting system, or whatever. Then have a helpdesk minion call the user in question and ask what they’re trying to do. Perhaps they’ve forgotten their password. Perhaps someone else got access to their account. Perhaps they’re having trouble. Maybe they need sudo -l explained to them.

The end user will either feel like you’re watching out for them, or realize that your sysadmin group watches the systems very closely.

Even if you don’t take proactive action, having sudo failures logged to a separate file simplifies digging through the logs.

Technology versus Democracy

Yesterday’s election was mostly a primary, but also included a few millage issues. The purpose of a primary is to keep the obvious maniacs from getting onto the final ballot, so I make the effort to vote. (Your definition of “obvious maniacs” probably differs from mine, but that’s okay.)

I’m waiting for verification, and am glad to see that they’ve finally replaced the big printed books with a laptop. But all of the verification people are standing around the laptop, getting more and more frustrated. One of them is on the phone. “Yes, we entered the password. No, it’s not letting us into the site. He’s entering it again. Yes, I’m sure the Caps Lock key is off. He’s trying again. Yes, the password we’re using is ‘election’.”

I’m third person in line. The line is growing quickly behind me. I peer at the laptop, lean in, and quietly say “Excuse me, but your Caps Lock light is on.”

The guy at the keyboard turns off Caps Lock, the password is entered, and the poll workers quickly get us all through. Everybody hails me as a technical genius. Which I might well be, if you define “genius” as “someone who looks to see WHICH LIGHTS ARE ON.”

I have three points to make on this seemingly pointless anecdote:

  • In discussions about electronic voting machines, remember: these are the people who have time and interest to work the polls. They have no awareness of good security practice. They don’t troubleshoot, because they don’t really know how to. It’s not a question of age: one of the people was a senior citizen, two were middle-aged, and one mid-twenties.
  • If you have time to volunteer as a poll worker, do so. Democracy isn’t about voting; it’s about doing things to help your community.
  • I am the savior of democracy. In Precinct Three, at least.
  • PS: I didn’t include the real password here, but the actual password was just as bad.

    FreeBSD: portmaster with pkgng

    I recently tried FreeBSD’s pkgng, based on Ivan Voras’ blog post. Days after getting the new machine set up, though, I got this in my daily status mail:


    Checking for packages with security vulnerabilities:
    Database fetched: Fri Aug 3 03:02:57 EDT 2012
    apache-2.2.22_5 is vulnerable:
    Apache -- Insecure LD_LIBRARY_PATH handling

    WWW: http://portaudit.FreeBSD.org/de2bc01f-dc44-11e1-9f4d-002354ed89bc.html

    php5-5.4.4 is vulnerable:
    php -- potential overflow in _php_stream_scandir

    WWW: http://portaudit.FreeBSD.org/bdab0acd-d4cd-11e1-8a1c-14dae9ebcf89.html

    Where did this come from? A bit of poking around my system leads me to /usr/local/etc/periodic/security/410.pkg-audit. My first question is, where did this file come from? pkgng includes an equivalent to the old pkg_info -W:

    $ pkg which /usr/local/etc/periodic/security/410.pkg-audit
    /usr/local/etc/periodic/security/410.pkg-audit was installed by package pkg-1.0.r4

    pkgng gives you an audit of your packages in the daily mail. Excellent. It’s a DragonFly feature that I really like, and long overdue in FreeBSD.

    So, how to upgrade the insecure ports? Unfortunately, I’ve had to build apache from source. I can’t use packages. That means I need to build the new version from ports. I get the new version ports tree with:

    # portsnap fetch extract

    I like to use portmaster for managing my ports. I’m told it works with pkgng. Let’s find out. First, tell ports that we’re running under pkgng by setting a variable in /etc/make.conf:

    WITH_PKGNG=yes

    Now install portmaster:

    # cd /usr/ports/ports-mgmt/portmaster
    # make all install clean

    But portmaster doesn’t seem to work with pkgng:

    $ portmaster -L
    ===>>> Root ports (No dependencies, not depended on)
    ===>>> 0 root ports

    ===>>> Trunk ports (No dependencies, are depended on)
    ===>>> 0 trunk ports

    ===>>> Branch ports (Have dependencies, are depended on)
    ===>>> 0 branch ports

    ===>>> Leaf ports (Have dependencies, not depended on)
    ===>>> 0 leaf ports

    ===>>> 0 total installed ports
    ===>>> There are no new versions available

    A bit of research shows that portmaster needs to be patched before it works with pkgng. (Portmaster has a large install base, the portmaster maintainer is very careful about not messing up existing users, and pkgng is still very young. I’m confident that portmaster will work with pkgng by the time pkgng becomes the default.) You can get the patch on github.

    Now build portmaster with the patch.

    # cd /usr/ports/ports-mgmt/portmaster/
    # make patch
    # cd work/portmaster-3.13.13
    # patch < $HOME/patch-portmaster-pkgng
    # cd ..
    # cd ..
    # make all install

    I keep the patch in my home directory, because any time I want to rebuild portmaster I must reapply the patch. And if the patch fails, I must check for a new patch.

    Try the new portmaster:

    # portmaster -L
    ===>>> Root ports (No dependencies, not depended on)
    ===>>> pkg-1.0.r4
    ===>>> New version available: pkg-1.0.r5_1
    ===>>> portmaster-3.13.13
    ...

    Much better.

    I’ll start by upgrading pkgng itself, then my other ports. I use:

    # portmaster -d --no-confirm pkg

    Use whatever portmaster options you prefer, of course. With the pkgng patch, portmaster seems to behave exactly as you expect. But the only way we’ll know for sure is if you test pkgng in your environment and file bug reports with the appropriate maintainer.

    Cisco radius auth for users and enable

    All authentication on my network (with carefully selected exceptions) should be centralized. This includes router administrative logins via telnet or SSH. My authentication information is in an OpenLDAP 2.4 server. Attaching Cisco gear to an OpenLDAP database is hard. But attaching Cisco gear to RADIUS is pretty easy. But my FreeRADIUS server uses LDAP as its back end, and attaching Cisco gear to RADIUS is pretty easy.

    To have your enable password, you’ll need an LDAP user called $enab15$. Take careful note of how that is spelled: dollar sign, ENAB, the number 15, and another dollar sign. There is no L, and no second E. Add this user to any LDAP groups needed for RADIUS access. This user’s password is your enable secret.

    Create a loopback address for the router.

    interface Loopback0
    description management
    ip address 192.0.2.13 255.255.255.255

    Use this address for all router management functions. If you use an IP for a real interface for management or monitoring, you can have trouble when that interface goes down.

    Then tell the router about your RADIUS servers. Always list multiple RADIUS servers. If you have only one RADIUS server, get a second, preferably on a completely different part of your network.

    ip radius source-interface Loopback0
    radius-server host 192.0.2.253 auth-port 1812 acct-port 1813 key RadiusSecret
    radius-server host 192.0.2.252 auth-port 1812 acct-port 1813 key RadiusSecret

    Create a local enable password and a local user with administrative privileges. These will come into play when your RADIUS servers fail. (Hopefully, they never will. But assuming things will go well sets yourself up for a really bad day.)

    enable secret 5 $1$lnds$LNrkh4d8aoeuY/Q2Akm1k7
    username admin privilege 15 password 7 1D1C1B050B8290E1

    Now attach the radius servers to the authentication system.

    aaa new-model
    aaa authentication login default group radius local
    aaa authentication enable default group radius enable

    With this setup, your Cisco will try radius first, and then fall back to the local authentication file if your RADIUS server does not answer.

    You might also need to attach your virtual terminals to your authentication settings.

    line vty 0 4
    login authentication default
    line vty 5 15
    login authentication default

    You should now have Cisco user names and the enable password synchronized with LDAP, through RADIUS.

    Debugging RANCID

    I’m a big fan of RANCID for managing configurations for embedded devices, such as most routers and switches. While you can go buy CiscoWorks, OpenView, or any number of proprietary products, RANCID is good enough for the overwhelming majority of us. (Those products do have other advantages, but simple configuration revision control isn’t one of them.)

    For those who haven’t used RANCID: it logs into your devices every hour, gets the device configuration, and compares it to the stored configuration. If the configuration has changed, RANCID checks the new version into CVS. Combined with CVSWeb, RANCID really simplifies embedded device management.

    Every now and then it breaks, however. Last week, I started getting an email every hour, whining that RANCID couldn’t get the configuration of one of my Mikrotik border routers. I hadn’t changed the router configuration in several days. My cow-orkers claimed they hadn’t touched the router.

    So, let’s see what RANCID is having trouble with.

    Log into the RANCID server, and su – to your RANCID account. Use clogin(1) to log into the device.

    %clogin edge-1
    edge-1
    spawn ssh -c 3des -x -l admin+ct edge-1
    admin+ct@edge-1.lodden.com's password:
    ...

    [admin@edge-1] >

    So, I can log in.

    The main command to get a Mikrotik configuration is export. I run the command. It completes, but takes a few minutes. Not really a shock — this device has several full BGP feeds on IPv4 and IPv6, packet filtering, traffic shaping, and folds my socks in its spare time.

    So, it’s not the obvious problem; the router can export its config, and RANCID can log into the router.

    So, run RANCID for the group that includes the trouble router.

    %rancid-run mikrotik

    %

    No error messages, but let’s check the log. It’s full of messages like this:

    ...
    Trying to get all of the configs.
    edge-1: End of run not found
    Error: TIMEOUT reached
    =====================================
    ...

    Well, that’s not good. Let’s try running a single command on the router, setting the timeout to the usual 90 seconds.

    %clogin -t 90 -c "export;quit" edge-1
    edge-1
    spawn ssh -c 3des -x -l admin+ct edge-1
    admin+ct@edge-1.lodden.com's password:
    ...
    /ipv6 nd prefix default
    set autonomous=yes preferred-lifetime=1w valid-lifetime=4w2d

    Error: TIMEOUT reached
    %

    So, the export takes longer to run than RANCID’s default timeout. How long does it need? Run RANCID under time(1) to find out. Add -t 1000 to set the timeout to 1000 seconds.

    % time clogin -t 1000 -c “export;quit” edge-1

    Walk away. Eventually, come back to look at it.

    ...
    set accounting=yes default-group=read exclude-groups="" interim-update=0s \
    use-radius=no
    [admin@edge-1] > quit
    interrupted
    Connection to edge-1.lodden.com closed.

    Error: EOF received
    0.102u 0.094s 2:57.84 0.1% 87+948k 0+0io 0pf+0w

    This export took almost three minutes, or 180 seconds. Twice the default timeout. Ick.

    Now we have to tell RANCID to use a different timeout. I didn’t find anything in the manual pages, so I asked on the rancid-discuss mailing list. John Heasley quickly answered. It seems that the timeout option in .cloginrc should cover this, but that the feature is missing from the Mikrotik login script. He included a patch. I applied the patch and added

    add password edge-1 blahblah
    add user edge-1 admin+ct
    add method edge-1 ssh
    add noenable edge-1 {1}
    add timeout edge-1 500

    I then re-run RANCID. It completes silently. I can’t be sure that the change actually works until I see RANCID check in a change. I logged into the router, corrected a typo in the login message, and tried again. This time, changes appeared in CVS and I received my email. So I can conclude the patch works.

    The most important thing to do in all of this, though? Close the loop with John Heasley. Verify the patch works, so others can benefit from my annoyance.

    On a related note, RANCID is one of those tools that gets less attention than it deserves. I’m pondering writing a short book about it, rather like SSH Mastery. Would anyone actually be interested, however?

    Configuring OpenBSD to use RADIUS auth

    I have a love-hate relationship with RADIUS. RADIUS is the cheap white glue of authentication. Just about everything speaks it, so you can use it as cheap glue to unify passwords across your gear. But it’s a finicky protocol, with lots of edge cases, and those edges can be SHARP.

    Okay, perhaps it’s more of a tolerate-hate relationship. But still.

    OpenBSD supports using RADIUS to authenticate user accounts. Why would you possibly want to do this? For one thing, if you’re using authpf, it gives you a way to easily synchronize firewall passwords with your Windows domain through the Microsoft Internet Authentication Service. It’s not ideal — ideal would be making all of your users use public key auth — but it’s better than nothing, reduces your support workload, and helps convince management that your firewall is a real solution.

    I’m assuming you already have a RADIUS server. Mine is freeRADIUS-2.1.12, but any basic RADIUS server will work.

    OpenBSD uses BSD authentication, a competitor to PAM. The RADIUS auth process is documented in login_radius(8), but I’ll walk through the basics.

    Configure your RADIUS server to allow access from your OpenBSD box, and assign it a shared secret. My RADIUS server is 192.0.2.2, and my secret is the string Insubordinate. (It’s a lousy secret, but it’s just an example, and I’m tired.)

    Create a directory for the server list, and set the permissions as per the man page.

    # mkdir /etc/raddb
    # chgrp _radius /etc/raddb/
    # chmod 755 /etc/raddb/

    Create the file /etc/raddb/servers. List each server, and its secret, on its own line.

    192.0.2.2 Insubordinate

    Now switch your default authentication scheme in login.conf.

    #auth-defaults:auth=passwd,skey:
    auth-defaults:\
            :auth=radius:\
            :radius-port=1812:\
            :radius-server=192.0.2.2:

    The change should take effect immediately. Be sure you have a window logged in as root beforehand, so you can switch back if necessary. If you have trouble, check your RADIUS server’s debugging log, or use a packet sniffer to examine the actual RADIUS dialog.

    Your host will now check every password against the RADIUS server. That includes root. You probably want to set up an auth-su class, or only have the authpf class use RADIUS, or some combination thereof. I really recommend requiring anyone who can get an actual shell to log in with public key authentication.

    And for those who care, you now know exactly what part of Absolute OpenBSD I’m writing.

    SSH course at BSDCan 2012

    I will be at BSDCan 2012. In addition to poking my nose where it doesn’t belong, spouting insouciance and irrelevance, and derailing important technical discussions with tediously pointless anecdotes about my pet rats, I am teaching a course on SSH.

    If you don’t have time to work your way through the SSH book, take a couple hours and and get dragged through it. The course will include material not included in the book, unless it goes too long, in which case I’ll spout off about extra material upon request.

    The new FreeBSD Device Drivers book should be out by then, and the author will be there. (Pity poor Joe Kong. He came to BSDCan in 2007, just after his Designing BSD Rootkits came out. I spent the rest of the conference badgering him into writing the FreeBSD Device Drivers book. He spent five years on this project, and is done at long last. I’m sure he’s ready to shout “Ha! Lucas, I’m rid of your stupid project! I’m a free man!” Sadly for Joe, I have another three books he needs to write. Preferably immediately. If he resists my persuasive powers, I’m prepared to use brainwashing techniques.)

    If you’re a reader of my blog, you need to go to BSDCan. We received so many excellent paper proposals this year that cutting the list down was physically painful. If you don’t yet have IPv6 up and running, attend the IPv6 tutorial. Or learn about PF. Not to mention two days of really interesting papers. And beer.

    Small server IPv4/IPv6 pf.conf

    I’m deploying IPv6 for my employer. While getting corporate servers up on IPv6 is nice and all, of course I put priority on my own personal Web server.

    Just because IPv6 is still populated mostly by early adopters, doesn’t mean we can neglect basic system security. That means that the server needs a packet filter for both IPv4 and IPv6. PF supports filtering both protocols in one ruleset.

    The following is a unified IPv4/IPv6 PF ruleset for a small server. It:

  • Allows all traffic from management addresses
  • Allows all ICMP and ICMPv6 traffic
  • Allows traffic to specific ports where we choose to provide service — in this case, SMTP, DNS, and HTTP.
  • Blocks everything else.

    Fill in the IPv4 and IPv6 external addresses, define your interface, adjust the permitted services to match your environment, and you’re ready to go.

    ext_if="em0"
    ext_addr="{192.0.2.40, 192.0.2.41}"
    ext_v6="2001:db8:0:12::2"

    table <mgmt_hosts> const {172.16.0.0/24, 172.16.5.0/24}
    table <v6_mgmt_hosts> const {2001:db8:1:4::2}

    set block-policy return
    set loginterface $ext_if
    set skip on lo0

    scrub in all no-df

    block in all

    pass in on $ext_if proto icmp all
    pass in on $ext_if proto icmp6 all

    pass in on $ext_if inet from <mgmt_hosts>
    pass in on $ext_if inet6 from <v6_mgmt_hosts>

    pass out on $ext_if inet from $ext_addr to any
    pass out on $ext_if inet6 from $ext_v6 to any

    #services we permit
    pass in on $ext_if proto tcp from any to $ext_addr port {25,53,80}
    pass in on $ext_if proto udp from any to $ext_addr port 53
    pass in on $ext_if inet6 proto tcp from any to $ext_v6 port {53,80}
    pass in on $ext_if inet6 proto udp from any to $ext_v6 port 53