Book Review: The Practice of Network Security Monitoring

Most computer books are badly written. The information in the book is fine (usually, hopefully), but the actual craft of writing is poor. They read like computer programs. This isn’t surprising, as most computer books are written by computer professionals. By the time you’re good enough at a computing topic to write a book about it, your brain automatically arranged things in machine-friendly order. That’s human nature. The downside of this, however, is that most computing books lack the things that make books interesting to human beings. We readers grit our teeth and plow through them because we need the information.

I’m pleased to say that Richard Bejtlich’s The Practice of Network Security Monitoring is not one of those books. The damn thing is actually readable. By normal people.

That’s a vague assertion. How about a metric? Season 6 of Burn Notice just hit Netflix streaming. I watched a few episodes Saturday. They ended on a tense cliffhanger, but I finally had to go to bed. Sunday, I finished reading this book before seeing how Westin and company got out of their fix. (Okay, that’s not exactly a metric, but it’s a good sign.)

Bejtlich graduated from Harvard and the Air Force Academy graduate. He led CIRT teams in the Air Force, built a security team at General Electric, and is now Chief Security Officer at Mandiant. He’s on television as an electronic security guru. And for the last decade-plus, he’s been beating the drum about intelligent attackers and the need for a holistic approach to security. When everybody else was going on about firewalls and antivirus and access controls and penetration testing, he wrote books like The Tao of Network Security Monitoring arguing that we need to think about network defense as an ongoing activity. He made absurd claims like “prevention eventually fails” and “there are smart people slowly breaking into your network,” lumping these into an overall practice called Network Security Monitoring.

Time has proved that he was right.

Books like Tao and Extrusion Detection had a lot about the business process of security. They had specific examples of how to respond to security incidents. Other books, like my own Network Flow Analysis, cover using a specific tool that’s usable in a NSM context. But there hasn’t been a good book on how to deploy real security monitoring in your organization, across all tools — and, just as importantly, how to get buy-in from the business side on this.

The Practice of Network Security Monitoring does all that and more.

The book starts with an overview of the NSM philosophy and practice, and what makes it different from the conventional “we respond to intrusions” perspective. He spends some time going over the Security Onion toolkit. For those readers not familiar with SO Security Onion is to security monitoring what PfSense is for firewalls — an integrated toolkit built atop a free operating system. You can build everything you need for NSM without Security Onion, but like PfSense, why bother?

Richard gives a brief overview of the various tools in SO, from Sguil to Bro to Snort to Xplico and on and on and on. While you can hook these tools together yourself so they operate more or less seamlessly, again, SO has done all the work for you.

The best part of the book, however, is where Bejtlich takes us through two security incidents. He uses various Security Onion tools to dissect the data from an intrusion response system alert. He backtracks both a client-side and a server-side intrusion, and shows how to accurately scope the intrusion. Was only one server broken into? What data was stolen? What action can you take in response?

What really makes this book work is that he humanizes the security events. Computing professionals think that their job is taking care of the machine. That’s incorrect. Their main job is to interface between human beings and the computer. Sometimes this takes the form of implementing a specification from a written document, or solving a bug, or figuring out why your SSL web site is running slowly. Maybe most of your professional skill lies in running the debugger. That’s fine, and your skill is admirable. But the reason you get paid is because you interact with other human beings.

Bejtlich pays attention to this human interface. The security incidents happen because people screw up. And they screw up in believable ways — I read the server compromise walkthrough and thought “This could be me.” (Actually, it probably has been me, I just didn’t know it.) Deploying network security monitoring takes hardware, which means you need money and staff. Bejtlich advises the reader on how to approach this conversation, using metrics that competent managers understand. His scenarios include discouragement and even fear. If you’ve ever worked in intrusion response, you know those emotions are very much a part of cleaning up.

But he shows you how to deal with those problems and the attendant emotions: with data.

He even demonstrates practical, real-world examples in how to get that data when the tools fail.

Humanizing a tech book is no easy task. Most authors fail, or don’t even try. But Bejtlich pulls it off. He applies “prevention eventually fails” to both the people and the software, and the result is both readable and useful.

Is this book perfect for me? No. The sections on how to install Security Onion are written so that Windows administrators can use them. I don’t need that level of detail. But the end result is that tPoNSM is usable by people unfamiliar with Unix-like systems, so I can’t really fault him for that.

I should add a caveat here. Richard Bejtlich likes my books. He’s said so. At very great length. Repeatedly. Even though I’ve misspelled his name. More than once. And now I’m reviewing one of his books. I am predisposed to like his work because it’s hard to dislike someone who likes you. But if this book wasn’t good, I wouldn’t bother to review it. I read far more books than I review, and I would much rather not write a review than write a negative review. And anyone familiar with my work can assure you that I do not suck up.

tPoNSM is useful for anyone interested in the security of their own network. Many of the tools can actually be used outside of a security context, to troubleshoot network and system problems. Deploying NSM not only means you can quickly identify, contain, and remediate intrusions, it gives you insight into the network as a whole. You might start off looking for intrusions, but you’ll end up with a more stable network as a side effect.

You can buy the book at any bookstore. If you want to reward the author, buy it directly from No Starch Press and use coupon code NSM101. You’ll get both the print and electronic versions, and Richard will get a couple extra dollars.

Now if you’ll excuse me, there’s another dozen or so episodes of Burn Notice that need watching.

next tech book: Sudo Mastery

Last weekend I amused myself by tweeting:

Stupid contest: give the title of the tech book I’ve just started writing. If correct, you get to make me a sandwich.

The answer is Sudo Mastery. Obviously. Although there were some amusing and hopeful alternative suggestions.

As with DNSSEC Mastery, I’m making the in-progress draft available for purchase. I did this with DNSSEC Mastery, and people seemed pleased. So, let’s try this again.

You can buy Sudo Mastery now for $7.99. You get access to the early drafts of the book, the version sent for tech review, and the final version. Incomplete drafts are in PDF format, because I can’t see anyone loading an incomplete book onto their e-reader. The finished book will be in PDF, epub, and mobi.

The in-progress version also includes various markup and reserved pages for physical layout, as well as whatever notes I make during the writing process. The version currently on the site includes the outline for the part of Chapter 3 that I haven’t written yet.

When the book is complete, I will raise the price to $9.99. Buying early gets you a 20% discount.

You can also choose to overpay for the book (or any title on the site) if you desire. Because some of you want to. If you’re trying to make a go of being a writer, rule number 1 is: when someone puts money in your hand, you take it and say “Thank you.” There’s even an option to just give me money without getting anything, because people have said that they want to do that.

I will announce new versions of the book via Twitter. You’ll get an update every few chapters. As it’s a Mastery book, they’re short chapters. I’ll announce major milestones, just as a complete manuscript or completed tech edits, here.

I’m doing this for a couple reasons. One, people liked it last time. I get paid early, which is always nice. Feedback is good. And I expect that, once again, only my hardcore fans will buy an incomplete book. Some people will look at this as a 20% discount for preorders, which is fine too.

When you buy the book doesn’t matter to me. Sales made via third-party ebookstores are better for my career. They book the book’s sales rank and increase the book’s visibility. But the only people who will be interested in this offer are those interested enough in my work to stalk me via my blog or Twitter and, frankly, there’s not really enough of you to directly impact my Amazon sales rank.

You all do impact my sales, mind you, but indirectly. Every time you tell someone that they need to read one of my books, every time you leave a positive review on a book, every time you slap your boss and say “Dammit, make the support guys read this book so they leave me alone,” you help me a great deal. And that support drives bookstore rankings.

But as far as my stupid contest went: the best answer by far came from Darrin Chandler, who said:

Liked Absolut OpenBSD, but have since switched to Svedka. The morning afterboot still hurts.

I’m still in pain from that one.

live DNSSEC talk

On August 10, at 12:30PM EDT, I’ll be doing a talk on “DNSSEC in 55 Minutes” for the Metro Detroit Linux Users Group. I do this sort of thing all the time, but this time will be a little different.

The talk will go out live via Google Hangouts on Air. You’ll be able to see it via my Google Plus account. I haven’t done this before, but I believe you’ll be able to ask questions in a chat window. And it will be archived on YouTube for public viewing.

As it’s a Linux user group meeting, I will of course wear my Absolut OpenBSD shirt. (I really need to get a matching Absolut FreeBSD shirt one day.)

FreeBSD pkgng vs custom ports

My Web server runs FreeBSD. Some security updates prompted me to upgrade my installed packages. Running this server entirely with packages isn’t possible, so I installed PHP 5 from ports. This machine uses pkgng.

When I ran pkg upgrade, however, my web site stopped working. The server itself started just fine, and it did quite well serving downloadable PHP code to clients. The problem was pretty obvious:

I originally compiled PHP 5 from ports so to get the Apache PHP module. Running pkg upgrade replaced my custom-built PHP with one from packages, so Apache no longer had a PHP module.

So: I want to upgrade from packages, but not upgrade PHP automatically. pkgng has a tool for this, pkg lock. You must give pkg lock a package name and confirm that you want to update this package.

# pkg lock php5
php5-5.4.17: lock this package? [y/N]: y
Locking php5-5.4.17
#

If you give a nonexistent package name, pkg exits silently. Which is kinder than calling you an idiot, I suppose.

# pkg lock php-5
#

Personally, I’d rather be told I’m an idiot (aka “no such package”), because when I’m tired I might interpret this as “package locked, everything is good.” But whatever.

Before uninstalling a locked port, either via pkg or ports, you must unlock the package.

To upgrade this server now, I do the following:

# portsnap fetch update
# pkg upgrade
# cd /usr/ports/lang/php5 && make
# pkg unlock php5
# make deinstall && make reinstall
# pkg lock php5

Done!

I’m told by people who should know that eventually pkgng will let me do this entirely with packages, but for now, this will do nicely.

From my experience, upgrading with pkgng is MUCH nicer than upgrading with pkg_add. All of the issues that drove me away from binary package upgrades have disappeared.

I should also note, however, that the BSDs will have the ports tree for the forseeable future. We need it. Some popular ports, such as nginx, have 77,371,252,455,336,267,181,195,264 possible combinations. The packaging team is not going to build 2^86 nginx packages. But you can build whichever exact version you need.

It seems that pkgng is actually taking FreeBSD binary packaging back towards being useful. Once there’s an official public repository of pkgng packages, you really should try it out.

(Update: Hat tip to Allan Jude for pointing out that 2^86 is an impressively big number when you multiply it out.)

convert FreeBSD to pkgng with Ansible

Ansible includes a module to manage FreeBSD packages, if you’re using the forthcoming pkgng packaging system. The Ansible module isn’t complete yet, but as Ansible is moving really quickly, I’m pretty confident their FreeBSD support will grow additional knobs. As pkgng is increasingly close to production, and the PC-BSD folks have generously offered their 64-bit pkgng repository available to the public, this seems like a good time to make the move.

But I’m not about to make this change manually. Bootstrapping pkgng isn’t difficult, but I have a great big heap of FreeBSD VMs and I have other things I’d like to accomplish this month. Therefore, I’m bootstrapping my ability to manage FreeBSD packages via Ansible, with Ansible.

Before starting, you need an Ansible server and a pkgng repo.

All of my FreeBSD servers run 9.1, updated via freebsd-update. If you come across this article years later, adjust accordingly.

My Ansible server runs OpenBSD, and the OpenBSD ansible package has problems managing anything other than OpenBSD. I generically recommend running Ansible out of git.

You also need a pkgng repository. The official repository is in closed testing, but many FreeBSD developers are using it successfully. PC-BSD has made their 64-bit repository available to all FreeBSD users. And many people have built their own repository. Thanks to my awesome Twitter stalkers minions followers, I have access to more than one private repository. This example assumes you’re using the 64-bit-only PC-BSD repository.

Configure the pkgng repo in pkg.site. I keep my FreeBSD configuration files in /home/ansible/freebsd/etc/, so I make a /home/ansible/freebsd/etc/pkg.site that contains only:

packagesite: http://pkg.cdn.pcbsd.org/9.1-RELEASE/amd64
PUBKEY: /usr/local/etc/pkg-pubkey.cert
PKG_CACHEDIR: /usr/local/tmp

I also need the current PC-BSD public key, saved as pkg-pubkey.cert

With these two files and an Ansible install, we’re ready to deploy on the Ansible group freebsd-test. Here’s the runbook.

---
- hosts: freebsd-test
  user: ansible
  sudo: yes

  tasks:
  - name: install pkg tools
    action: command  pkg_add -r pkg
#do you need a proxy? Put it here
#    environment:
#      ftp_proxy: http://proxy.michaelwlucas.com:8080

  - name: edit /etc/make.conf
    action: shell echo "WITH_PKGNG=YES" >> /etc/make.conf

  - name: convert package database
    action: shell pkg2ng

#I have typed pkg_add for 18 years, and my fingers no longer listen to
#my brain. Disable pkg_* commands for safety
  - name: disable pkg_ commands
    action: shell chmod -x /usr/sbin/pkg_*

  - name: install pkg.conf
    action: copy src=/home/ansible/freebsd/etc/pkg.conf
      dest=/usr/local/etc/pkg.conf owner=root group=wheel mode=0644

#skip this if you're using a non-PCBSD repo
  - name: install pc-bsd pgp key
    action: copy src=/home/ansible/freebsd/etc/pkg-pubkey.cert
      dest=/usr/local/etc/pkg-pubkey.cert owner=root group=wheel mode=0644

#ansible pkg does not have upgrade command yet
#use shell to trigger upgrade
#pkgng package in pkg-old is always out of date, upgrade it
  - name: upgrade pkg pkg
    action: command pkg upgrade -qy 
#do you need a proxy? Put it here
#    environment:
#      ftp_proxy: http://proxy.michaelwlucas.com:8080

This takes a while to run.

Before deploying, test. Test again. And run your conversion in batches, so that you don’t scramble several hundred virtual machines simultaneously. Because that would really suck. Fortunately, by changing the group at the top of the playbook or specifying a new inventory file, you can batch these changes easily.

“Absolute OpenBSD” auction photographs

I promised to post photographs of the signed copy of Absolute OpenBSD 2nd Edition that went to Bill Allaire.

Here they are. In full size, so that Bill can later use them to authenticate his copy, in the extraordinarily unlikely event that he desires to do so. (And also so that perhaps the bandwidth utilized might finally exceed that generated by my fame in the gay porn world.)

All the developers signed the first page:
ao2e inside page

People immediately checked the index for their favorite features. Leave it to Bob Beck to find my extraordinarily subtle joke on BUFCACHEPERCENT, which I assure you I completely deliberately chose to do and was not at all an actual error.

ao2e index

Some developers weren’t terribly confident of their English skills. But that’s okay.
ao2e japanese

Sadly, there were some problems with the book. People were kind enough to point them out.
ao2e small margin

Many pages didn’t get marked up, but a few got extra attention. We now know exactly how Theo feels about file flags.
ao2e 174-175

Finally, Theo was kind enough to add a footnote. He didn’t mark it from the main body of the text, however. Programmers might have mad skillz, but footnotes are much harder than they look.
ao2e footnote page

As an aside: I considered getting the developers to sign my personal copy of the book while I was there. But that would reduce the uniqueness of this artifact, dang it. So Bill gets the awesome copy and the rest of us get pix.

“DNSSEC Mastery” business numbers

When SSH Mastery came out, I published the initial sales figures and a followup a month later. The results of publishing one book is not really data, however. It’s one data point.

A second data point is also not data. But it’s twice as close to data as one data point.

Let’s compare and contrast SSH Mastery with the book I just published a couple months ago, DNSSEC Mastery.

DNSSEC Mastery was pre-released via LeanPub, so that my hard-core fans could get copies of the incomplete draft. This was also an experiment. Now that the book is out on mainstream platforms such as Amazon and Kobo, LeanPub sales have basically gone to zero. (I still make an occasional sale thanks to LeanPub not requiring a PayPal account and being platform agnostic.) I’ve sold a total of 61 copies on LeanPub, most for the list price of $9.99.

The ebook was published 17 April 2013. Here are sales for the trailing end of April.

Here’s April’s DNSSEC Mastery sales:

  • Amazon Kindle: 13 books sold (7 US, 1 UK, 2 DE, 2 FR, 1 BR)
  • Barnes & Noble: 1 book sold
  • Smashwords: 1 book sold
  • Total sales: 15 books

    In May, the book had been out for a while. Also, the print book came out near the end of the month. What happened to sales?

  • Amazon Kindle: 32 books sold (26 US, 4 DE, 1 CA, 1 ES)
  • Barnes & Noble: 2 books sold
  • Smashwords: 4 books sold
  • TWP (direct from me): 4 books sold
  • Print: 23 books
  • Total sales: 42 ebooks, 23 print

    Then there’s June. Where did sales go:

  • Amazon Kindle: 17 books sold (11 US, 1 UK, 2 DE, 1 FR, 1 CA, 1 BR)
  • Barnes & Noble: zero
  • Smashwords: 1 book sold
  • TWP (direct from me): 6 ebooks sold
  • Print: 28 books sold, most at the beginning of the month
  • Total sales: 24 ebooks, 28 print

    June’s sales are down across all platforms.

  • Total total sales for first 3 months: 81 ebook, 51 print. Compared to the initial 2 months of sales on SSH Mastery, that’s pretty pathetic.

    What conclusions can I draw from these numbers?

    First, let’s talk about “fan base.” The story is, you need a fan base to pimp your book for you. Word of mouth is a powerful thing, true. But the key here is the word “need.” You can certainly be successful at self-publishing without an existing fan base.

    My fans are awesome. All of the LeanPub and TWP sales are from my devoted readers who specifically want books from me and want to support my writing. I appreciate every one of you. (I also think that you’re slightly daft for buying an incomplete book that hasn’t undergone any review whatsoever and might well be chock-full of dangerous advice, but that’s a separate issue.) Some of my fans waited until the book was officially released to purchase. I certainly don’t begrudge them this. Buying a book from your preferred cloud provider has distinct advantages, and that’s my personal preference as well. These readers are responsible for May’s surge of ebook sales and June’s higher print sales.

    Over 60 people bought copies of the book from LeanPub or TWP before the book was even complete. These hard-core fans would have presumably bought the book on a big-name commercial ebook platform instead. Selling the book through alternative channels deprived me of that initial “big sales burst” that so many self-publishing authors covet. That big initial sales burst doesn’t matter. The book is selling better than I expected. Less than I hoped, mind you, but better than I expected.

    I’m now done selling to the hard-core fans who stalk me on Twitter. Extra posts on Facebook and Twitter and whatever will not get me more readers. Instead, I’m selling to the public. And the public is buying.

    Would I like every single DNS administrator to deploy DNSSEC, using my book? Sure.

    But here’s the thing: if I continue to sell roughly 50 books a month to the general public, I will have more sales to the public than to my hard-core fans. I’m not discounting the fans, mind you — the initial influx of cash certainly helps, and their encouragement keeps me writing technology books. But in the long run, the book itself is the thing.

    What about “breaking even?” The expenses for SSH Mastery were very high, but I broke even in the first few months. What about breaking even on DNSSEC Mastery?

    Total expenses were about $650, between cover art, layout, editing, more editing, proof shipping, and such. (This does not include the various beers I owe assorted people.) I haven’t totaled all of the income across all the platforms, and I have no intention of doing that until the end of the year, but it’s pretty clear that this book has broken even with expenses. The only thing I need to get paid for is my time in writing the book. If the book continues to sell, then I’ll be okay on that front.

    The obvious question people will have is, “If you expected sales to be mediocre, why not write something with wider appeal?”

    I wrote DNSSEC Mastery as another test.

    At a technical level, SSH Mastery was very easy to write. I tested absolutely everything with my OpenBSD desktop, my Windows laptop, and a remote virtual machine. Nothing required interacting with anything outside my little world. Also, it’s very easy to see if my documentation worked. My usual testing methods of “capricious malicious play” work well with SSH.

    DNSSEC is different. It demands interacting with the entire world. This makes DNSSEC actually somewhat dangerous to play with. You can make your domains disappear from the Internet. You can make your users unable to reach the Internet. It’s an ugly, hairy thing to capriciously and maliciously play with.

    Additionally, DNSSEC is hard. My DNS knowledge dates from the mid-1990s. I was aware it had changed over the years, but I didn’t know exactly how or why. Writing this book meant updating my knowledge base.

    The community knowledge base on DNSSEC is spotty, unreliable, incoherent, and actively wrong. While folks started working on DNSSEC in the last century, it was only really finished in 2006. The years between generated a whole bunch of obsolete documentation on weird problems that no longer have any bearing on reality.

    Finally, DNSSEC has a really bad reputation. “It’s hard!” It’s another layer of complexity! It’s a whole chain of failures waiting to happen!” “There’s better ways to do this!”

    I don’t write books about torturing yourself. With the latest version of BIND, it’s entirely possible for your average overworked network administrator to deploy DNSSEC in the real world, without relying on overcomplicated third-party add-ons and scripts and random hacks. The software now handles the tedious and constant maintenance. This means that I could write a book about it in clear conscience.

    So: can I write and self-publish a technologically challenging book and have it be useful and correct? Can such a book solve problems for the reader and actually improve their organization?

    In that regard, DNSSEC Mastery is an unqualified success.