managing sshd with Ansible

My environment has two common tasks when managing OpenSSH servers: copying user’s authorized_keys files to the server, and changing the sshd configuration file /etc/ssh/sshd_config. I use Ansible for both, using a single playbook. Running the playbook updates all the authorized_keys files on every host and verifies that sshd is properly configured. (Not that any of my minions would reconfigure sshd without going through change control, or anything like that.)

I’ll start with authorized_keys management.

My servers are grouped by functions. Not all users have access to all servers, and I only want to copy a user’s authorized_keys file to a server if that user can access that server. Start by creating a group_vars directory at the top level of your Ansible install. Create a file for each group that you manage authorized_keys on. Within that file, create a YAML list of all users who can access your server. For example, my DNS servers are all in the Ansible group “dns.” I’ve created the file /etc/ansible/group_vars/dns, and it contains:

---

#users who get SSH access to these machines
sshusers:
  - mwlucas
  - ansible
  - john
  - harry
  - mandy

I have a similar file for my LDAP servers, in the group “ldap.” The file is /etc/ansible/group_vars/ldap.

---

#users who get SSH access to these machines
sshusers:
  - mwlucas
  - ansible
  - jake
  - harry

I think you see the trend here. Remember, YAML lists are space-sensitive, and tabs are forbidden.

You cannot define these lists in /etc/ansible/hosts: you must use a variables file.

Then there’s the SSH configuration part. I run an artisan network. (“Artisan network” means “each server is set up uniquely, managed uniquely, and has little in common with anything else, really frustrating any hopes of easy or consistent systems administration.” While I’m slowly dragging everything towards mass manageability, it’s not there yet.) Each host has multiple IP addresses. I want sshd to only listen on a single IP address. I can’t consistently pull the IP from the host itself, so I need to hard-code it in Ansible.

Fortunately, Ansible should already have this information. A host’s ssh daemon probably isn’t listening on the IP address that DNS gives for the host — that is, the IP referred to by the hostname ldap.michaelwlucas.com probably doesn’t have an SSH server on it. It might be ldap-ssh or ldap-mgmt or ldap-pleasekillmenow. So I’ve had to tell Ansible the address to connect to.

The directory /etc/ansible/host_vars contains a file for each host. The file /etc/ansible/host_vars/ldap.michaelwlucas.com contains something like this:

---

ansible_ssh_host: 192.0.2.88

The ansible_ssh_host is a dedicated Ansible variable, giving the hostname or IP address that Ansible should use to SSH into this host. I’m going to piggyback this for my SSH configuration file. (If you use hostnames in your ansible_ssh_host, you’ll need to either create a separate variable with the IP or to convert the hostname to an IP address somewhere in this process. If you figure out an easy way to do this, do let me know.)

So, how do I want sshd configured?

As I’m mass-managing these machines’ sshd service, I neither need nor want all the various default options in sshd_config. Having those options in the configuration file is great when you’re manually configuring sshd, but I don’t want anyone on any of these servers to change the sshd configuration without going through change control.

So, let’s go to a machine that has a properly configured sshd and get the non-default options.

$ grep -v ^# /etc/ssh/sshd_config > sshd_config.j2

The .j2 indicates this is a Jinja2 template, which is what Ansible uses for creating files. Look at what’s left after I remove the blank lines.

ListenAddress 192.0.2.88
PermitRootLogin without-password
AuthorizedKeysFile /etc/ssh/authorized_keys/%u
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
Subsystem sftp /usr/libexec/sftp-server

The only problem here is the ListenAddress setting, which needs to be different on each host. I make that a variable. I also add a revision tag and an Ansible management statement.

#$Id$
#{{ ansible_managed }}
ListenAddress {{ ansible_ssh_host }}
PermitRootLogin without-password
AuthorizedKeysFile /etc/ssh/authorized_keys/%u
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
Subsystem sftp /usr/libexec/sftp-server

The {{ ansible_ssh_host }} is a variable statement, pulling in that variable from Ansible itself.

So, how to use all this in a playbook?

---
- hosts: freebsd
  user: ansible
  sudo: yes

  tasks:
  - name: create key directory
    action: file path=/etc/ssh/authorized_keys state=directory
      owner=0 group=0 mode=0755

  - name: upload user key
    action: copy src=/home/ansible/crossplatform/etc/ssh/authorized_keys/{{ item }}
      dest=/etc/ssh/authorized_keys/
      owner=0 group=0 mode=644
    with_items: sshusers

  - name: sshd configuration file update
    template: src=/etc/ansible/configs/etc/ssh/sshd_config.j2
      dest=/etc/ssh/sshd_config
      backup=yes
      owner=0 group=0 mode=0644
      validate='/usr/sbin/sshd -T -f %s'
    notify:
    - restart sshd

  handlers:
    - name: restart sshd
      service: name=sshd state=restarted

The first task creates the directory for key storage. I do not allow users to upload authorized_keys files for their own account. We don’t want an intruder to add their own key to a user account. Instead, each user’s authorized keys are in a file named after the username, in the directory /etc/ssh/authorized_keys, and our sshd_config tells sshd to look in that directory.

The second task copies the user keys listed in the sshusers variable defined in the group_vars file. While Ansible has an authorized_keys module specifically for handling these files, it has problems with quotes in restricted keys. Until that’s fixed, I’ll fall back to the perfectly adequate “copy” module.

The third task reads the jinja2 template for sshd_config, adds the necessary information, and copies the file to the server. It also validates that the configuration is legitimate — not that it will do what you want, mind you, but it will verify that sshd understands this sshd_config file.

Last, we restart sshd.

My next steps with this will be to update the template so it works on non-FreeBSD hosts. But that’ll be a topic for another day.

Want more on configuring SSH? Check out my book SSH Mastery.

Command-Line FreeBSD Configuration: sysrc

The traditional BSD standard of “edit /etc/rc.conf” isn’t sustainable across large numbers of machines. If you must change dozens of servers you want a reliable way to alter the system without either manually editing every configuration file or some sed/awk hackery. (Running a sed/awk script to edit rc.conf on every server I own makes me nervous. I don’t do nervous these days.) FreeBSD 9.2 and later includes sysrc, a program to consistently and safely alter /etc/rc.conf and friends from the command line. (On older versions of FreeBSD, sysrc is available in ports.) I find sysrc very useful for Ansible-managed farms, but it should work just as well with Puppet or Chef or any configuration management system.

Start using sysrc by asking it what it knows about the system configuration.

$ sysrc -a
defaultrouter: 192.0.2.129
dumpdev: AUTO
hostname: mwltest3
ifconfig_em0: inet 192.0.2.160 netmask 255.255.255.128
keymap: us.dvorak.kbd
sshd_enable: YES

Oddly enough, this is exactly what’s in my rc.conf, minus all the comments and such.

I must enable ntpd(8) on this machine. Here, sysrc looks an awful lot like sysctl.

# sysrc ntpd_enable=YES
ntpd_enable: NO -> YES

Note that this doesn’t actually start ntpd, it merely enables it in the configuration. You must run /etc/rc.d/ntpd start or service ntpd start to start the daemon.

To disable a daemon, do the same thing in reverse.

# sysrc ntpd_enable=NO
ntpd_enable: YES -> NO

One potential problem with sysrc is that it’s a tool for editing /etc/rc.conf, not for configuring FreeBSD. It has no idea what legitimate values are. This means that if you make a typo, it propagates to rc.conf.

# sysrc ntpd_enable=YSE
ntpd_enable: NO -> YSE

Here sysrc works as advertised, but ntpd won’t. And you can arbitrarily enable nonexistent services.

# sysrc gerbil_enable=YES
gerbil_enable: -> YES

I do not have gerbils installed. But if I ever do install them, the gerbil wheel will start spinning without any further intervention.

I suspect that one day sysrc will grow a service integrity checker, but it solves my immediate needs.

Experienced FreeBSD administrators who run a small number of servers don’t need sysrc, but those of us with farms will find it invaluable. My thanks to Devin Teske for shepherding this into base.

Wanted: interesting sudoers

I’ve learned a lot about sudo while writing Sudo Mastery. One of the things I’ve learned is that many, many people have insecure sudo policies. Most tutorials, mine included, leave holes people who understand sudo can get through. I’ve also learned that many people are using sudo much more cleverly than I previously thought.

Sudo is perhaps the most widely used access control tool for Unix-like systems. I’d like this book to be accurate and useful. As such, I have a favor to ask my readers:

If you’re using sudo in production, and your sudoers file is pleasant and elegant, or it cleverly solves an tricky access problem, or it’s a horrible ghastly nightmare but you don’t know any other way to express the policy, I’d like you to send me a sanitized copy of your sudoers file.

I’m especially interested in “default deny” policies, where the word ALL doesn’t appear in the command field.

Don’t include real usernames or IP addresses.

And don’t send me anything you’re uncomfortable sharing.

I won’t cut-and-paste your policies, and anything I use will be further anonymized. But the world of sudo is huge, and there’s very little really good examples out there. The more good policies I read, the better the book will be.

You can email them to me at mwlucas at michael w lucas dotcom. Please use the word sudoers in the subject.

Thank you for your help.

Book Review: The Practice of Network Security Monitoring

Most computer books are badly written. The information in the book is fine (usually, hopefully), but the actual craft of writing is poor. They read like computer programs. This isn’t surprising, as most computer books are written by computer professionals. By the time you’re good enough at a computing topic to write a book about it, your brain automatically arranged things in machine-friendly order. That’s human nature. The downside of this, however, is that most computing books lack the things that make books interesting to human beings. We readers grit our teeth and plow through them because we need the information.

I’m pleased to say that Richard Bejtlich’s The Practice of Network Security Monitoring is not one of those books. The damn thing is actually readable. By normal people.

That’s a vague assertion. How about a metric? Season 6 of Burn Notice just hit Netflix streaming. I watched a few episodes Saturday. They ended on a tense cliffhanger, but I finally had to go to bed. Sunday, I finished reading this book before seeing how Westin and company got out of their fix. (Okay, that’s not exactly a metric, but it’s a good sign.)

Bejtlich graduated from Harvard and the Air Force Academy graduate. He led CIRT teams in the Air Force, built a security team at General Electric, and is now Chief Security Officer at Mandiant. He’s on television as an electronic security guru. And for the last decade-plus, he’s been beating the drum about intelligent attackers and the need for a holistic approach to security. When everybody else was going on about firewalls and antivirus and access controls and penetration testing, he wrote books like The Tao of Network Security Monitoring arguing that we need to think about network defense as an ongoing activity. He made absurd claims like “prevention eventually fails” and “there are smart people slowly breaking into your network,” lumping these into an overall practice called Network Security Monitoring.

Time has proved that he was right.

Books like Tao and Extrusion Detection had a lot about the business process of security. They had specific examples of how to respond to security incidents. Other books, like my own Network Flow Analysis, cover using a specific tool that’s usable in a NSM context. But there hasn’t been a good book on how to deploy real security monitoring in your organization, across all tools — and, just as importantly, how to get buy-in from the business side on this.

The Practice of Network Security Monitoring does all that and more.

The book starts with an overview of the NSM philosophy and practice, and what makes it different from the conventional “we respond to intrusions” perspective. He spends some time going over the Security Onion toolkit. For those readers not familiar with SO Security Onion is to security monitoring what PfSense is for firewalls — an integrated toolkit built atop a free operating system. You can build everything you need for NSM without Security Onion, but like PfSense, why bother?

Richard gives a brief overview of the various tools in SO, from Sguil to Bro to Snort to Xplico and on and on and on. While you can hook these tools together yourself so they operate more or less seamlessly, again, SO has done all the work for you.

The best part of the book, however, is where Bejtlich takes us through two security incidents. He uses various Security Onion tools to dissect the data from an intrusion response system alert. He backtracks both a client-side and a server-side intrusion, and shows how to accurately scope the intrusion. Was only one server broken into? What data was stolen? What action can you take in response?

What really makes this book work is that he humanizes the security events. Computing professionals think that their job is taking care of the machine. That’s incorrect. Their main job is to interface between human beings and the computer. Sometimes this takes the form of implementing a specification from a written document, or solving a bug, or figuring out why your SSL web site is running slowly. Maybe most of your professional skill lies in running the debugger. That’s fine, and your skill is admirable. But the reason you get paid is because you interact with other human beings.

Bejtlich pays attention to this human interface. The security incidents happen because people screw up. And they screw up in believable ways — I read the server compromise walkthrough and thought “This could be me.” (Actually, it probably has been me, I just didn’t know it.) Deploying network security monitoring takes hardware, which means you need money and staff. Bejtlich advises the reader on how to approach this conversation, using metrics that competent managers understand. His scenarios include discouragement and even fear. If you’ve ever worked in intrusion response, you know those emotions are very much a part of cleaning up.

But he shows you how to deal with those problems and the attendant emotions: with data.

He even demonstrates practical, real-world examples in how to get that data when the tools fail.

Humanizing a tech book is no easy task. Most authors fail, or don’t even try. But Bejtlich pulls it off. He applies “prevention eventually fails” to both the people and the software, and the result is both readable and useful.

Is this book perfect for me? No. The sections on how to install Security Onion are written so that Windows administrators can use them. I don’t need that level of detail. But the end result is that tPoNSM is usable by people unfamiliar with Unix-like systems, so I can’t really fault him for that.

I should add a caveat here. Richard Bejtlich likes my books. He’s said so. At very great length. Repeatedly. Even though I’ve misspelled his name. More than once. And now I’m reviewing one of his books. I am predisposed to like his work because it’s hard to dislike someone who likes you. But if this book wasn’t good, I wouldn’t bother to review it. I read far more books than I review, and I would much rather not write a review than write a negative review. And anyone familiar with my work can assure you that I do not suck up.

tPoNSM is useful for anyone interested in the security of their own network. Many of the tools can actually be used outside of a security context, to troubleshoot network and system problems. Deploying NSM not only means you can quickly identify, contain, and remediate intrusions, it gives you insight into the network as a whole. You might start off looking for intrusions, but you’ll end up with a more stable network as a side effect.

You can buy the book at any bookstore. If you want to reward the author, buy it directly from No Starch Press and use coupon code NSM101. You’ll get both the print and electronic versions, and Richard will get a couple extra dollars.

Now if you’ll excuse me, there’s another dozen or so episodes of Burn Notice that need watching.

next tech book: Sudo Mastery

Last weekend I amused myself by tweeting:

Stupid contest: give the title of the tech book I’ve just started writing. If correct, you get to make me a sandwich.

The answer is Sudo Mastery. Obviously. Although there were some amusing and hopeful alternative suggestions.

As with DNSSEC Mastery, I’m making the in-progress draft available for purchase. I did this with DNSSEC Mastery, and people seemed pleased. So, let’s try this again.

You can buy Sudo Mastery now for $7.99. You get access to the early drafts of the book, the version sent for tech review, and the final version. Incomplete drafts are in PDF format, because I can’t see anyone loading an incomplete book onto their e-reader. The finished book will be in PDF, epub, and mobi.

The in-progress version also includes various markup and reserved pages for physical layout, as well as whatever notes I make during the writing process. The version currently on the site includes the outline for the part of Chapter 3 that I haven’t written yet.

When the book is complete, I will raise the price to $9.99. Buying early gets you a 20% discount.

You can also choose to overpay for the book (or any title on the site) if you desire. Because some of you want to. If you’re trying to make a go of being a writer, rule number 1 is: when someone puts money in your hand, you take it and say “Thank you.” There’s even an option to just give me money without getting anything, because people have said that they want to do that.

I will announce new versions of the book via Twitter. You’ll get an update every few chapters. As it’s a Mastery book, they’re short chapters. I’ll announce major milestones, just as a complete manuscript or completed tech edits, here.

I’m doing this for a couple reasons. One, people liked it last time. I get paid early, which is always nice. Feedback is good. And I expect that, once again, only my hardcore fans will buy an incomplete book. Some people will look at this as a 20% discount for preorders, which is fine too.

When you buy the book doesn’t matter to me. Sales made via third-party ebookstores are better for my career. They book the book’s sales rank and increase the book’s visibility. But the only people who will be interested in this offer are those interested enough in my work to stalk me via my blog or Twitter and, frankly, there’s not really enough of you to directly impact my Amazon sales rank.

You all do impact my sales, mind you, but indirectly. Every time you tell someone that they need to read one of my books, every time you leave a positive review on a book, every time you slap your boss and say “Dammit, make the support guys read this book so they leave me alone,” you help me a great deal. And that support drives bookstore rankings.

But as far as my stupid contest went: the best answer by far came from Darrin Chandler, who said:

Liked Absolut OpenBSD, but have since switched to Svedka. The morning afterboot still hurts.

I’m still in pain from that one.

live DNSSEC talk

On August 10, at 12:30PM EDT, I’ll be doing a talk on “DNSSEC in 55 Minutes” for the Metro Detroit Linux Users Group. I do this sort of thing all the time, but this time will be a little different.

The talk will go out live via Google Hangouts on Air. You’ll be able to see it via my Google Plus account. I haven’t done this before, but I believe you’ll be able to ask questions in a chat window. And it will be archived on YouTube for public viewing.

As it’s a Linux user group meeting, I will of course wear my Absolut OpenBSD shirt. (I really need to get a matching Absolut FreeBSD shirt one day.)

FreeBSD pkgng vs custom ports

My Web server runs FreeBSD. Some security updates prompted me to upgrade my installed packages. Running this server entirely with packages isn’t possible, so I installed PHP 5 from ports. This machine uses pkgng.

When I ran pkg upgrade, however, my web site stopped working. The server itself started just fine, and it did quite well serving downloadable PHP code to clients. The problem was pretty obvious:

I originally compiled PHP 5 from ports so to get the Apache PHP module. Running pkg upgrade replaced my custom-built PHP with one from packages, so Apache no longer had a PHP module.

So: I want to upgrade from packages, but not upgrade PHP automatically. pkgng has a tool for this, pkg lock. You must give pkg lock a package name and confirm that you want to update this package.

# pkg lock php5
php5-5.4.17: lock this package? [y/N]: y
Locking php5-5.4.17
#

If you give a nonexistent package name, pkg exits silently. Which is kinder than calling you an idiot, I suppose.

# pkg lock php-5
#

Personally, I’d rather be told I’m an idiot (aka “no such package”), because when I’m tired I might interpret this as “package locked, everything is good.” But whatever.

Before uninstalling a locked port, either via pkg or ports, you must unlock the package.

To upgrade this server now, I do the following:

# portsnap fetch update
# pkg upgrade
# cd /usr/ports/lang/php5 && make
# pkg unlock php5
# make deinstall && make reinstall
# pkg lock php5

Done!

I’m told by people who should know that eventually pkgng will let me do this entirely with packages, but for now, this will do nicely.

From my experience, upgrading with pkgng is MUCH nicer than upgrading with pkg_add. All of the issues that drove me away from binary package upgrades have disappeared.

I should also note, however, that the BSDs will have the ports tree for the forseeable future. We need it. Some popular ports, such as nginx, have 77,371,252,455,336,267,181,195,264 possible combinations. The packaging team is not going to build 2^86 nginx packages. But you can build whichever exact version you need.

It seems that pkgng is actually taking FreeBSD binary packaging back towards being useful. Once there’s an official public repository of pkgng packages, you really should try it out.

(Update: Hat tip to Allan Jude for pointing out that 2^86 is an impressively big number when you multiply it out.)

convert FreeBSD to pkgng with Ansible

Ansible includes a module to manage FreeBSD packages, if you’re using the forthcoming pkgng packaging system. The Ansible module isn’t complete yet, but as Ansible is moving really quickly, I’m pretty confident their FreeBSD support will grow additional knobs. As pkgng is increasingly close to production, and the PC-BSD folks have generously offered their 64-bit pkgng repository available to the public, this seems like a good time to make the move.

But I’m not about to make this change manually. Bootstrapping pkgng isn’t difficult, but I have a great big heap of FreeBSD VMs and I have other things I’d like to accomplish this month. Therefore, I’m bootstrapping my ability to manage FreeBSD packages via Ansible, with Ansible.

Before starting, you need an Ansible server and a pkgng repo.

All of my FreeBSD servers run 9.1, updated via freebsd-update. If you come across this article years later, adjust accordingly.

My Ansible server runs OpenBSD, and the OpenBSD ansible package has problems managing anything other than OpenBSD. I generically recommend running Ansible out of git.

You also need a pkgng repository. The official repository is in closed testing, but many FreeBSD developers are using it successfully. PC-BSD has made their 64-bit repository available to all FreeBSD users. And many people have built their own repository. Thanks to my awesome Twitter stalkers minions followers, I have access to more than one private repository. This example assumes you’re using the 64-bit-only PC-BSD repository.

Configure the pkgng repo in pkg.site. I keep my FreeBSD configuration files in /home/ansible/freebsd/etc/, so I make a /home/ansible/freebsd/etc/pkg.site that contains only:

packagesite: http://pkg.cdn.pcbsd.org/9.1-RELEASE/amd64
PUBKEY: /usr/local/etc/pkg-pubkey.cert
PKG_CACHEDIR: /usr/local/tmp

I also need the current PC-BSD public key, saved as pkg-pubkey.cert

With these two files and an Ansible install, we’re ready to deploy on the Ansible group freebsd-test. Here’s the runbook.

---
- hosts: freebsd-test
  user: ansible
  sudo: yes

  tasks:
  - name: install pkg tools
    action: command  pkg_add -r pkg
#do you need a proxy? Put it here
#    environment:
#      ftp_proxy: http://proxy.michaelwlucas.com:8080

  - name: edit /etc/make.conf
    action: shell echo "WITH_PKGNG=YES" >> /etc/make.conf

  - name: convert package database
    action: shell pkg2ng

#I have typed pkg_add for 18 years, and my fingers no longer listen to
#my brain. Disable pkg_* commands for safety
  - name: disable pkg_ commands
    action: shell chmod -x /usr/sbin/pkg_*

  - name: install pkg.conf
    action: copy src=/home/ansible/freebsd/etc/pkg.conf
      dest=/usr/local/etc/pkg.conf owner=root group=wheel mode=0644

#skip this if you're using a non-PCBSD repo
  - name: install pc-bsd pgp key
    action: copy src=/home/ansible/freebsd/etc/pkg-pubkey.cert
      dest=/usr/local/etc/pkg-pubkey.cert owner=root group=wheel mode=0644

#ansible pkg does not have upgrade command yet
#use shell to trigger upgrade
#pkgng package in pkg-old is always out of date, upgrade it
  - name: upgrade pkg pkg
    action: command pkg upgrade -qy 
#do you need a proxy? Put it here
#    environment:
#      ftp_proxy: http://proxy.michaelwlucas.com:8080

This takes a while to run.

Before deploying, test. Test again. And run your conversion in batches, so that you don’t scramble several hundred virtual machines simultaneously. Because that would really suck. Fortunately, by changing the group at the top of the playbook or specifying a new inventory file, you can batch these changes easily.