Sudo Mastery off to copyeditor

I just shipped the tech-reviewed copy of Sudo Mastery off to the copyeditor. She’ll have it back to me in a few days, and the book will move into production immediately thereafter.

This means that the pre-order discount will expire soon. How soon is soon? It’s soon.

Now I’m off to work on one of my other 2013 goals. Thanks to my appendix’s untimely detonation at the beginning of the year and my Europe trip I won’t accomplish everything on that list, but that’s no reason to not get as many of them finished as possible.

EuroBSDCon, and Sudo Mastery

How’s that for diverse topics in one post?

I just got back from EuroBSDCon 2013 on Malta. The EuroBSDCon Foundation and Andre Opperman did a great job with the conference, and presented one of the best sets of talks and keynotes and related programs I’ve seen in years. It’s motivated me to try to improve BSDCan, but I’ll babble about that in another post.

I followed EuroBSDCon with a few days in Paris, to talk to other authors, network, and figure out some “business of writing” stuff. Plus see the Eiffel Tower and the catacombs, of course.

Now that I’m home, I’m diving into the technical reviews of Sudo Mastery.

Normally when I have a book out for tech review, I post a variety of reminders during the time people should be reviewing. “Two weeks to get comments back to me!” “One week!” “Six hours and three minutes!” I didn’t do that this time, instead focusing on things like distributing blacklists via BGP and automated deployment of FreeBSD and Bhyve and relayd and and and and…

In a weird coincidence, I haven’t received as many tech reviews as I usually do.

Why do people nag? Because it works.

If you’re one of the folks who volunteered to review the manuscript, you have a couple days left to send me comments. I would really like to get the book to the copyeditor by next Monday.

“Sudo Mastery” tech reviewers wanted

Thursday night, I finished the first draft of Sudo Mastery. Today, I went through the manuscript, removed my known tics, discovered a few more tics that needed killing, cleaned up bits and pieces, and now have a work ready for technical review.

Which is where you lot come in. I’m looking for people with sudo experience to read the book and tell me where I’ve screwed up. My screw-ups usually come in two flavors:

1) I’m technically wrong.
2) I use something in a way other people don’t
3) I don’t include something important, because I’ve never used it.

The goal of Sudo Mastery is not to get 100% of my readers to 100% sudo expertise, but instead to get 90% of my readers everything they will need. The remaining 10% will get a solid grounding in sudo and pointers on solving their particularly pernicious edge cases. The idea is roughly similar to my other Mastery books or Cisco Routers for the Desperate.

The contents of Sudo Mastery are:

  1. Introduction
  2. sudo and sudoers
  3. editing and testing sudoers
  4. lists and aliases
  5. options and defaults
  6. shell escapes, editors, and sudoers policies
  7. configuring sudo
  8. user environments versus sudo
  9. sudo for intrusion detection
  10. sudoers distribution and complex policies
  11. security policies in ldap
  12. logging & debugging
  13. authentication

Most of these chapters are short. And much of the writing needs rewriting. But there’s no point in rewriting until I know the content is technically correct.

If you know sudo, if you consider yourself a sudo master already, this is your chance to spread your wisdom. Read my general notes for tech reviewers, and email me at mwlucas at michael w lucas dotcom. (The W is vastly important… you might get a response from the domain without one, but it won’t be what you expected.)

I plan to send out manuscripts over the next week. I’m asking for people to return their comments on or before 5 October. I plan to revise the manuscript the week of 6 October and get it to the copyeditor before the 15th.

With anything resembling luck, the completed book will be available before Thanksgiving. I’d really like to have the holidays off this year.

First draft of “Sudo Mastery” complete

I just typed the last words of the first draft of Sudo Mastery.

The completed first draft is available for early purchasers. As it’s no longer an incomplete draft, I’ve raised the early purchase price to $8.99. That’s more than the really early buyers paid, but less than the final price. (Selling the early drafts from my own bookstore lets me experiment, so I’m ratcheting up the price to see what happens.)

What happens now?

First, I take a couple days and do something else. Anything else. This is vital, as I need some distance from the manuscript. I know it’s a big steaming pile of bodily waste, sure. But I need to be able to see the details of how, exactly, that pile is arranged.

Then: go over the manuscript from beginning to end, looking for obvious technical and writing problems.

Then spellcheck the book. (The purpose of an as-you-type spellchecker is to slow down the writing process. Note that a grammar checker never enters into this process.)

Then solicit technical reviewers. (Don’t volunteer yet: if you do, I’ll put you on my list of people who can’t follow directions.)

Then I go to EuroBSDCon. When I return, I integrate the comments into the book in another round of testing and fact-checking and rewriting.

Off to copyeditor.

Fix what the copyeditor finds.

Then the book comes out.

I’m not writing an Ansible book…

…at least not now.

This is a “post it now so I can point to it later” piece.

I met Michael DeHaan, the Ansible creator, primary author, lead, and probably Grand Poobah, at AnsibleFest in Boston. We discussed the possibility of an Ansible book. He’s certainly open to the idea.

But we agreed that Ansible is moving too dang quickly to document in a book. By the time I finished a book, progress in Ansible would make the book obsolete. Ansible development will slow down at some time, making a book much more realistic.

I’m also not convinced that I’m the right person to write this book. I use a narrow slice of Ansible features, and other folks use a much greater set of Ansible features. My use is expanding, but still, I’m more likely to write a series of small Ansible books that reflect my growing understanding as opposed to one massive tome.

I’ll continue to document what I learn, and we’ll see what the future holds.

Cross-platform OpenSSH server management with Ansible

I’ve previously written about managing the OpenSSH server with Ansible. That example focused on my BSD servers. I also manage Ubuntu and CentOS machines as well as my FreeBSD and OpenBSD. While the BSD machines are very similar, Ubuntu and CentOS might are two different operating systems. Can I manage all of them by hand? Sure. But some of these call their SSH service ssh, while others call it sshd. They store their SFTP servers in different directories.

I want to manage all of these simultaneously. With one script. When I need to change my sshd configuration, I want the change to propagate across the network, to all of my machines, simultaneously.

Ansible makes it pretty easy to write a single sshd configuration that works on all systems, through templates. A template is a Jinja2 file that fills in variables. Jinja2 is closely related to Python, but it isn’t Python.

The first thing you must do is decide what how you want your SSH server configured. Most operating systems ship a very inclusive example sshd_config, with most options commented out out. But to see how your SSH server is actually configured, run sshd -T.

# sshd -T
port 22
protocol 2
addressfamily any
listenaddress [::]:22
listenaddress 0.0.0.0:22
usepam 1
serverkeybits 1024
logingracetime 120
keyregenerationinterval 3600
x11displayoffset 10

This prints out the running sshd configuration, with all the options as the running instance uses them.

Generate this configuration across your operating systems. Compare them. Which settings do you really want? What differences don’t matter in your environment? Which differences are set in your server by default? (If you aren’t familiar with the innards of OpenSSH configuration, permit me to suggest a book on the subject.)

Some operating systems set their default sshd_config options differently. For example, FreeBSD’s default is UsePam yes. CentOS and Ubuntu default to UsePam no. I’m safest if I specify UsePAM in my configuration file, to avoid ambiguity.

But OpenBSD doesn’t use PAM. If I use that configuration item in an sshd_config for that machine, then sshd won’t run.

Similarly, there’s the location of the sftp-server program. Different operating systems store these helper programs in different locations.

The way to handle all of these is to use a group variables, and assign the things that change a variable. Here’s my new cross-platform template for sshd_config.

#{{ ansible_managed }}
Protocol 2
ListenAddress {{ ansible_ssh_host }}
X11Forwarding yes

PubkeyAuthentication yes
AuthorizedKeysFile /etc/ssh/authorized_keys/%u
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
{% if ansible_system == 'OpenBSD' %}
#no PAM on OpenBSD
{% else %}
UsePAM yes
{% endif %}

Subsystem sftp {{ program_sftp_server }}

Each per-operating-system group file defines the variable program_sftp_server. Ansible automatically populates the variable ansible_system with the operating system name. And ansible_ssh_host comes from a per-host variable.

I would rather use something like “unless ansible_system=’OpenBSD'” to print UsePAM in the configuration file for the operating systems that need it, but that doesn’t seem to be an option in Jinja2.

The playbook changes a little as well:

---
- hosts: managed-sshd
user: ansible
sudo: yes

tasks:
- name: create key directory
action: file path=/etc/ssh/authorized_keys state=directory
owner=0 group=0 mode=0755

- name: upload user key
action: copy src=/home/ansible/crossplatform/etc/ssh/authorized_keys/{{ item }}
dest=/etc/ssh/authorized_keys/
owner=0 group=0 mode=644
with_items: sshusers

- name: sshd configuration file update
template: src=/home/ansible/crossplatform/etc/ssh/common-sshd_config.j2
dest=/etc/ssh/sshd_config
backup=yes
owner=0 group=0 mode=0644
validate='/usr/sbin/sshd -T -f %s'
notify:
- restart sshd

handlers:
- name: restart sshd
service: name={{ service_sshd }} state=restarted

The key difference here is that I’ve defined a per-OS variable, service_sshd. The operating systems can call their sshd servers sshd, ssh, or derangedweasel for all I care. I figure it out once, set it in the variable file, and get on with my life.

managing sshd with Ansible

My environment has two common tasks when managing OpenSSH servers: copying user’s authorized_keys files to the server, and changing the sshd configuration file /etc/ssh/sshd_config. I use Ansible for both, using a single playbook. Running the playbook updates all the authorized_keys files on every host and verifies that sshd is properly configured. (Not that any of my minions would reconfigure sshd without going through change control, or anything like that.)

I’ll start with authorized_keys management.

My servers are grouped by functions. Not all users have access to all servers, and I only want to copy a user’s authorized_keys file to a server if that user can access that server. Start by creating a group_vars directory at the top level of your Ansible install. Create a file for each group that you manage authorized_keys on. Within that file, create a YAML list of all users who can access your server. For example, my DNS servers are all in the Ansible group “dns.” I’ve created the file /etc/ansible/group_vars/dns, and it contains:

---

#users who get SSH access to these machines
sshusers:
  - mwlucas
  - ansible
  - john
  - harry
  - mandy

I have a similar file for my LDAP servers, in the group “ldap.” The file is /etc/ansible/group_vars/ldap.

---

#users who get SSH access to these machines
sshusers:
  - mwlucas
  - ansible
  - jake
  - harry

I think you see the trend here. Remember, YAML lists are space-sensitive, and tabs are forbidden.

You cannot define these lists in /etc/ansible/hosts: you must use a variables file.

Then there’s the SSH configuration part. I run an artisan network. (“Artisan network” means “each server is set up uniquely, managed uniquely, and has little in common with anything else, really frustrating any hopes of easy or consistent systems administration.” While I’m slowly dragging everything towards mass manageability, it’s not there yet.) Each host has multiple IP addresses. I want sshd to only listen on a single IP address. I can’t consistently pull the IP from the host itself, so I need to hard-code it in Ansible.

Fortunately, Ansible should already have this information. A host’s ssh daemon probably isn’t listening on the IP address that DNS gives for the host — that is, the IP referred to by the hostname ldap.michaelwlucas.com probably doesn’t have an SSH server on it. It might be ldap-ssh or ldap-mgmt or ldap-pleasekillmenow. So I’ve had to tell Ansible the address to connect to.

The directory /etc/ansible/host_vars contains a file for each host. The file /etc/ansible/host_vars/ldap.michaelwlucas.com contains something like this:

---

ansible_ssh_host: 192.0.2.88

The ansible_ssh_host is a dedicated Ansible variable, giving the hostname or IP address that Ansible should use to SSH into this host. I’m going to piggyback this for my SSH configuration file. (If you use hostnames in your ansible_ssh_host, you’ll need to either create a separate variable with the IP or to convert the hostname to an IP address somewhere in this process. If you figure out an easy way to do this, do let me know.)

So, how do I want sshd configured?

As I’m mass-managing these machines’ sshd service, I neither need nor want all the various default options in sshd_config. Having those options in the configuration file is great when you’re manually configuring sshd, but I don’t want anyone on any of these servers to change the sshd configuration without going through change control.

So, let’s go to a machine that has a properly configured sshd and get the non-default options.

$ grep -v ^# /etc/ssh/sshd_config > sshd_config.j2

The .j2 indicates this is a Jinja2 template, which is what Ansible uses for creating files. Look at what’s left after I remove the blank lines.

ListenAddress 192.0.2.88
PermitRootLogin without-password
AuthorizedKeysFile /etc/ssh/authorized_keys/%u
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
Subsystem sftp /usr/libexec/sftp-server

The only problem here is the ListenAddress setting, which needs to be different on each host. I make that a variable. I also add a revision tag and an Ansible management statement.

#$Id$
#{{ ansible_managed }}
ListenAddress {{ ansible_ssh_host }}
PermitRootLogin without-password
AuthorizedKeysFile /etc/ssh/authorized_keys/%u
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
Subsystem sftp /usr/libexec/sftp-server

The {{ ansible_ssh_host }} is a variable statement, pulling in that variable from Ansible itself.

So, how to use all this in a playbook?

---
- hosts: freebsd
  user: ansible
  sudo: yes

  tasks:
  - name: create key directory
    action: file path=/etc/ssh/authorized_keys state=directory
      owner=0 group=0 mode=0755

  - name: upload user key
    action: copy src=/home/ansible/crossplatform/etc/ssh/authorized_keys/{{ item }}
      dest=/etc/ssh/authorized_keys/
      owner=0 group=0 mode=644
    with_items: sshusers

  - name: sshd configuration file update
    template: src=/etc/ansible/configs/etc/ssh/sshd_config.j2
      dest=/etc/ssh/sshd_config
      backup=yes
      owner=0 group=0 mode=0644
      validate='/usr/sbin/sshd -T -f %s'
    notify:
    - restart sshd

  handlers:
    - name: restart sshd
      service: name=sshd state=restarted

The first task creates the directory for key storage. I do not allow users to upload authorized_keys files for their own account. We don’t want an intruder to add their own key to a user account. Instead, each user’s authorized keys are in a file named after the username, in the directory /etc/ssh/authorized_keys, and our sshd_config tells sshd to look in that directory.

The second task copies the user keys listed in the sshusers variable defined in the group_vars file. While Ansible has an authorized_keys module specifically for handling these files, it has problems with quotes in restricted keys. Until that’s fixed, I’ll fall back to the perfectly adequate “copy” module.

The third task reads the jinja2 template for sshd_config, adds the necessary information, and copies the file to the server. It also validates that the configuration is legitimate — not that it will do what you want, mind you, but it will verify that sshd understands this sshd_config file.

Last, we restart sshd.

My next steps with this will be to update the template so it works on non-FreeBSD hosts. But that’ll be a topic for another day.

Want more on configuring SSH? Check out my book SSH Mastery.

Command-Line FreeBSD Configuration: sysrc

The traditional BSD standard of “edit /etc/rc.conf” isn’t sustainable across large numbers of machines. If you must change dozens of servers you want a reliable way to alter the system without either manually editing every configuration file or some sed/awk hackery. (Running a sed/awk script to edit rc.conf on every server I own makes me nervous. I don’t do nervous these days.) FreeBSD 9.2 and later includes sysrc, a program to consistently and safely alter /etc/rc.conf and friends from the command line. (On older versions of FreeBSD, sysrc is available in ports.) I find sysrc very useful for Ansible-managed farms, but it should work just as well with Puppet or Chef or any configuration management system.

Start using sysrc by asking it what it knows about the system configuration.

$ sysrc -a
defaultrouter: 192.0.2.129
dumpdev: AUTO
hostname: mwltest3
ifconfig_em0: inet 192.0.2.160 netmask 255.255.255.128
keymap: us.dvorak.kbd
sshd_enable: YES

Oddly enough, this is exactly what’s in my rc.conf, minus all the comments and such.

I must enable ntpd(8) on this machine. Here, sysrc looks an awful lot like sysctl.

# sysrc ntpd_enable=YES
ntpd_enable: NO -> YES

Note that this doesn’t actually start ntpd, it merely enables it in the configuration. You must run /etc/rc.d/ntpd start or service ntpd start to start the daemon.

To disable a daemon, do the same thing in reverse.

# sysrc ntpd_enable=NO
ntpd_enable: YES -> NO

One potential problem with sysrc is that it’s a tool for editing /etc/rc.conf, not for configuring FreeBSD. It has no idea what legitimate values are. This means that if you make a typo, it propagates to rc.conf.

# sysrc ntpd_enable=YSE
ntpd_enable: NO -> YSE

Here sysrc works as advertised, but ntpd won’t. And you can arbitrarily enable nonexistent services.

# sysrc gerbil_enable=YES
gerbil_enable: -> YES

I do not have gerbils installed. But if I ever do install them, the gerbil wheel will start spinning without any further intervention.

I suspect that one day sysrc will grow a service integrity checker, but it solves my immediate needs.

Experienced FreeBSD administrators who run a small number of servers don’t need sysrc, but those of us with farms will find it invaluable. My thanks to Devin Teske for shepherding this into base.

Wanted: interesting sudoers

I’ve learned a lot about sudo while writing Sudo Mastery. One of the things I’ve learned is that many, many people have insecure sudo policies. Most tutorials, mine included, leave holes people who understand sudo can get through. I’ve also learned that many people are using sudo much more cleverly than I previously thought.

Sudo is perhaps the most widely used access control tool for Unix-like systems. I’d like this book to be accurate and useful. As such, I have a favor to ask my readers:

If you’re using sudo in production, and your sudoers file is pleasant and elegant, or it cleverly solves an tricky access problem, or it’s a horrible ghastly nightmare but you don’t know any other way to express the policy, I’d like you to send me a sanitized copy of your sudoers file.

I’m especially interested in “default deny” policies, where the word ALL doesn’t appear in the command field.

Don’t include real usernames or IP addresses.

And don’t send me anything you’re uncomfortable sharing.

I won’t cut-and-paste your policies, and anything I use will be further anonymized. But the world of sudo is huge, and there’s very little really good examples out there. The more good policies I read, the better the book will be.

You can email them to me at mwlucas at michael w lucas dotcom. Please use the word sudoers in the subject.

Thank you for your help.

Book Review: The Practice of Network Security Monitoring

Most computer books are badly written. The information in the book is fine (usually, hopefully), but the actual craft of writing is poor. They read like computer programs. This isn’t surprising, as most computer books are written by computer professionals. By the time you’re good enough at a computing topic to write a book about it, your brain automatically arranged things in machine-friendly order. That’s human nature. The downside of this, however, is that most computing books lack the things that make books interesting to human beings. We readers grit our teeth and plow through them because we need the information.

I’m pleased to say that Richard Bejtlich’s The Practice of Network Security Monitoring is not one of those books. The damn thing is actually readable. By normal people.

That’s a vague assertion. How about a metric? Season 6 of Burn Notice just hit Netflix streaming. I watched a few episodes Saturday. They ended on a tense cliffhanger, but I finally had to go to bed. Sunday, I finished reading this book before seeing how Westin and company got out of their fix. (Okay, that’s not exactly a metric, but it’s a good sign.)

Bejtlich graduated from Harvard and the Air Force Academy graduate. He led CIRT teams in the Air Force, built a security team at General Electric, and is now Chief Security Officer at Mandiant. He’s on television as an electronic security guru. And for the last decade-plus, he’s been beating the drum about intelligent attackers and the need for a holistic approach to security. When everybody else was going on about firewalls and antivirus and access controls and penetration testing, he wrote books like The Tao of Network Security Monitoring arguing that we need to think about network defense as an ongoing activity. He made absurd claims like “prevention eventually fails” and “there are smart people slowly breaking into your network,” lumping these into an overall practice called Network Security Monitoring.

Time has proved that he was right.

Books like Tao and Extrusion Detection had a lot about the business process of security. They had specific examples of how to respond to security incidents. Other books, like my own Network Flow Analysis, cover using a specific tool that’s usable in a NSM context. But there hasn’t been a good book on how to deploy real security monitoring in your organization, across all tools — and, just as importantly, how to get buy-in from the business side on this.

The Practice of Network Security Monitoring does all that and more.

The book starts with an overview of the NSM philosophy and practice, and what makes it different from the conventional “we respond to intrusions” perspective. He spends some time going over the Security Onion toolkit. For those readers not familiar with SO Security Onion is to security monitoring what PfSense is for firewalls — an integrated toolkit built atop a free operating system. You can build everything you need for NSM without Security Onion, but like PfSense, why bother?

Richard gives a brief overview of the various tools in SO, from Sguil to Bro to Snort to Xplico and on and on and on. While you can hook these tools together yourself so they operate more or less seamlessly, again, SO has done all the work for you.

The best part of the book, however, is where Bejtlich takes us through two security incidents. He uses various Security Onion tools to dissect the data from an intrusion response system alert. He backtracks both a client-side and a server-side intrusion, and shows how to accurately scope the intrusion. Was only one server broken into? What data was stolen? What action can you take in response?

What really makes this book work is that he humanizes the security events. Computing professionals think that their job is taking care of the machine. That’s incorrect. Their main job is to interface between human beings and the computer. Sometimes this takes the form of implementing a specification from a written document, or solving a bug, or figuring out why your SSL web site is running slowly. Maybe most of your professional skill lies in running the debugger. That’s fine, and your skill is admirable. But the reason you get paid is because you interact with other human beings.

Bejtlich pays attention to this human interface. The security incidents happen because people screw up. And they screw up in believable ways — I read the server compromise walkthrough and thought “This could be me.” (Actually, it probably has been me, I just didn’t know it.) Deploying network security monitoring takes hardware, which means you need money and staff. Bejtlich advises the reader on how to approach this conversation, using metrics that competent managers understand. His scenarios include discouragement and even fear. If you’ve ever worked in intrusion response, you know those emotions are very much a part of cleaning up.

But he shows you how to deal with those problems and the attendant emotions: with data.

He even demonstrates practical, real-world examples in how to get that data when the tools fail.

Humanizing a tech book is no easy task. Most authors fail, or don’t even try. But Bejtlich pulls it off. He applies “prevention eventually fails” to both the people and the software, and the result is both readable and useful.

Is this book perfect for me? No. The sections on how to install Security Onion are written so that Windows administrators can use them. I don’t need that level of detail. But the end result is that tPoNSM is usable by people unfamiliar with Unix-like systems, so I can’t really fault him for that.

I should add a caveat here. Richard Bejtlich likes my books. He’s said so. At very great length. Repeatedly. Even though I’ve misspelled his name. More than once. And now I’m reviewing one of his books. I am predisposed to like his work because it’s hard to dislike someone who likes you. But if this book wasn’t good, I wouldn’t bother to review it. I read far more books than I review, and I would much rather not write a review than write a negative review. And anyone familiar with my work can assure you that I do not suck up.

tPoNSM is useful for anyone interested in the security of their own network. Many of the tools can actually be used outside of a security context, to troubleshoot network and system problems. Deploying NSM not only means you can quickly identify, contain, and remediate intrusions, it gives you insight into the network as a whole. You might start off looking for intrusions, but you’ll end up with a more stable network as a side effect.

You can buy the book at any bookstore. If you want to reward the author, buy it directly from No Starch Press and use coupon code NSM101. You’ll get both the print and electronic versions, and Richard will get a couple extra dollars.

Now if you’ll excuse me, there’s another dozen or so episodes of Burn Notice that need watching.