Diagnosing “+Limiting icmp unreach response from…” with tcpdump

Anyone who has run a FreeBSD server for any length of time has seen these messages in their daily security emails. (You do read those, right?)

+Limiting icmp unreach response from 296 to 200 packets/sec
+Limiting icmp unreach response from 337 to 200 packets/sec
+Limiting icmp unreach response from 318 to 200 packets/sec
+Limiting icmp unreach response from 535 to 200 packets/sec
+Limiting icmp unreach response from 332 to 200 packets/sec
+Limiting icmp unreach response from 328 to 200 packets/sec

Way back in the Bronze Age, I learned that this mean “someone is port scanning.” The usual advice is to disable these messages by setting the sysctl net.inet.icmp.icmplim to 0. This silences the messages. I’m guilty of giving that advice myself.

What it really means is that something is sending your server UDP packets on a port that isn’t open. This could be a port scanner. It could also be a host legitimately trying to reach your host for a service it thinks you provide, or a service your host should be providing but isn’t.

I could go to my netflow collector and run a few commands to track down where these packets are coming from. In this case, the problem host is my netflow collector. I’m somewhat leery of using a tool to diagnose itself. An initial check shows that everything on the collector is running, so let’s see if it’s still happening with tcpdump.

I could run tcpdump -i em0 icmp and see all the ICMP traffic, but that’s inelegant. I don’t want to miss the traffic I’m looking for amidst a torrent of ICMP. And why have my brain filter traffic when ICMP will do it for me?

The first step is to identify exactly what we’re looking for. ICMP isn’t a monolithic protocol. Where TCP and UDP have ports, ICMP has types and codes. You can find a friendly list of types and codes here, or my readers can look in my Network Flow Analysis.

ICMP’s “port unreachable” message is type 3, code 3. Unlike TCP ports, the type and code are separate fields. Type 3 is “destination unreachable,” while the code indicates exactly what is unreachable — the port, the network, whatever. Type is ICMP field 0, while code is ICMP field 1. Tcpdump lets you filter on these just like the more familiar port numbers. Enclose more complicated filter expressions in quotes.

# tcpdump -ni em0 "icmp[0]=3 and icmp[1]=3"
10:01:03.287063 IP 10.250.250.10 > 192.0.2.214: ICMP 10.250.250.10 udp port 11022 unreachable, length 36
10:01:03.331388 IP 10.250.250.10 > 192.0.2.214: ICMP 10.250.250.10 udp port 11022 unreachable, length 36
10:01:03.356052 IP 10.250.250.10 > 192.0.2.214: ICMP 10.250.250.10 udp port 11022 unreachable, length 36
10:01:03.378256 IP 10.250.250.10 > 192.0.2.214: ICMP 10.250.250.10 udp port 11022 unreachable, length 36
10:01:03.411046 IP 10.250.250.10 > 192.0.2.214: ICMP 10.250.250.10 udp port 11022 unreachable, length 36
10:01:03.437458 IP 10.250.250.10 > 192.0.2.214: ICMP 10.250.250.10 udp port 11022 unreachable, length 36
10:01:03.457858 IP 10.250.250.10 > 192.0.2.214: ICMP 10.250.250.10 udp port 11022 unreachable, length 36

The host 192.0.2.214 is constantly trying to reach my collector on port 11022. 192.0.2.214 is my busiest border router.

That’s a router. This is a netflow collector. Maybe it’s netflow traffic? Let’s see.

# tcpdump -ni em0 -T cnfp ip host 192.0.2.214 and udp port 11022
192.0.2.214.11022 > 10.250.250.10.11022: NetFlow v5, 1897575.270 uptime, 1363184870.488773000, #1285199613, 30 recs
started 1897571.570, last 1897571.570
...

Yep. Either my router or my collector is misconfigured. And my monitoring system is misconfigured, because it should have caught that the collector process isn’t running. Or I should have noticed that I wasn’t actually getting any flow files from the collector running on another port.

Now to go back in time, find that young punk who wrote Absolute BSD, and whup his butt.

Some “Absolute OpenBSD 2/e” dates

No Starch intends to send AO2e to the printer on 22 March 2013. This would give a “bound book date” of approximately 12 April. Books would be in their hands roughly 19 April. They’re really good about shipping books to purchasers as soon as possible.

Note that DNSSec Mastery should be available in ebook form about then. Not only do I have two books coming in 2013, I have two books coming in April 2013.

All dates are subject to change based on the whim of the printer, phase of the moon, gasoline shortages, insurrections and iniquity and incivility, or any other reason whatsoever.

Any Firefox add-on people out there?

I’ve had really good luck asking random people to do work for me, so I’m going to try it again.

RFC6698 defines the DANE protocol for attaching information to DNSSEC-secured DNS. Notably, you can validate SSL certificates via DNS. This is a game-changer. The key here is the TLSA DNS record.

Web browsers don’t support this yet, but there is the Extended DNSSEC Validator Firefox add-on at os3sec.com, with source at github.

If you have the newest version of the add-on installed, sites like https://www.dnssec.michaelwlucas.com/ show up as secure. There is no “invalid certificate” warning. That’s because I’ve published a TLSA record for this zone, telling the browser that the certificate with fingerprint such-and-such on this port on this host is trustworthy. (Install the plugin from the github source, not from the xpi on the site.)

The interesting thing about this add-on is that it uses libunbound to perform DNSSEC validation at the client. Your local DNS servers don’t need to support DNSSEC. All you need to hack on this plugin is a desktop.

But the add-on doesn’t support BSD — it’s Linux, MS, and Mac only. The add-on authors don’t have time for BSD support, but gave me a couple hints on how to implement it. The plugin can’t find libunbound on BSD.

That seems like it would be easy to do. I’m capable of building the add-on from source, but I’ve never programmed any add-ons before. The source code looks like it’s easy to hack, but my efforts all segfaulted Firefox. Obviously, I need more expertise.

So, if you know anything about Firefox add-ons, or have ever wanted to hack on them, this is your chance.

DANE and TLSA are the killer applications for DNSSEC. The ability to validate cryptographic certificates via DNS is a game changer. (Cliche, but true.) You can have separate certificates for separate ports on a host. With DANE, there’s no longer any need for a self-signed certificate to be a disadvantage.

my OpenSSH AuthorizedKeysCommand script

The bleeding-edge OpenSSH supports using an AuthorizedKeysCommand statement in sshd_config to get the authorized_keys file for a user. This lets you store your authorized_keys files in LDAP, but avoids linking OpenSSH against OpenLDAP. (You could actually use any data store for your back end, but LDAP is both the most popular and what I have.)

Your AuthorizedKeysCommands script should take one argument, and return a series of authorized keys, one per line. CentOS has a script, which I previously mentioned, and one of my readers was kind enough to put together an OpenBSD port for it.

But there’s some problems with the CentOS approach, from a cross-platform perspective. While it’s fine on CentOS, porting and installing it on other platforms is more difficult than it needs to be. I’d really like something that has very few dependencies, is easy to install, and is easily portable across platforms.

When setting up a couple of Ubuntu boxes I came across an Ubuntu AuthorizedKeysCommand script. His approach is much simpler than the CentOS approach, but his script didn’t work for me out of the box, and it wouldn’t work on other operating systems without modification. But it inspired me to write my own script, one that works across all of my operating systems, using only the tools I already have on all of my servers. My results follow. But before I offer my script, I feel obliged to warn you.

The bad news is, I am not a programmer. I have been told by multiple independent parties that my code makes the Baby Jesus cry.

The really bad with this is that you need an ldapsearch command that returns your public keys, and this command is hard-coded with this script. The ideal script would read the system ldap config, but you’d probably have to hard-code a path to ldap.conf anyway.

The appalling news is, I wrote it in Perl. Why? Well, the evidence indicates that I am insane. But every server in my environment runs Perl, and it’s been ten years since I used enough sed/awk to make this work. A proper authentication credentials script would not use Perl.

I’m not suggesting that you take my code. I’m hoping to motivate one of my programmer readers, whose code only makes the Baby Jesus grimace a little, to do better. I mean, I’ve set a pretty low bar here.


#!/usr/bin/perl
#takes one argument, the username to get the keys for
die unless $ARGV[0];

open (LDAP, "/usr/bin/ldapsearch -L -xZb\"dc=michaelwlucas,dc=com\" '(&(objectClass=posixAccount)(uid=$ARGV[0]))' sshPublicKey |") || die "ldapsearch failed $!\n";

while (<LDAP>) {
next if /^#|^version|^dn\:|^\s*$/;
s/\n//;
s/\://g;
s/sshPublicKey/\n/;

s/^ //;
print;
}
print "\n";

One catch with this is that you’re processing the raw output of ldapsearch. An acceptable ldapsearch will return keys that look something like this:


...
sshPublicKey: ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAw9zmtbk8bT+7OVpVvzuYYQqRhF8j
...blah, blah, blah...
ilXzBw== rsa-key-20110114

Initially, my LDAP searches returned some keys that looked like this:
...
sshPublicKey:: c3NoLWRzcyANCkFBQUFCM056YUMxa2MzTUFBQUNCQVBhZmNhemdmUHI1T29DdFp
...blah, blah, blah...
y5sb2NhbA==

This key doesn’t start with ssh-rsa or ssh-dss, and it doesn’t even look like a SSH key. But if you look at the key in a graphical tool like phpLDAPAdmin, it has the leading ssh-rsa. In my case, the key had an extra carriage return at the end of the key, which meant that ldapsearch had to base64 encode it. Get rid of extra whitespace. The LPK patch doesn’t notice that space, but a simple external script does.

Of course, if someone wanted to write an AuthorizedKeysCommand that was portable, handled that encoding, and didn’t suck as bad as mine, it wouldn’t hurt my feelings at all. Really.

Conference Submissions & BSDCan 2013

The BSDCan 2013 call for proposals is open. As a BSDCon committee member, it’s my responsibility to get you lot to submit interesting papers. So: submit!

More than once, I’ve been asked something like “How can I get a paper accepted at BSDCan?” or “Why was my BSDCan paper rejected?” Here’s my answer to that general question. Lots of this applies to any conference, but as I’m not on the committees for those other conferences, I can’t claim any authority there. Other conferences have their own ideas, and other BSDCan committee members might have independent thoughts as well.

The purpose of a BSDCan paper is not to advertise the presenter’s work, but to put butts in seats. Sometimes these overlap. I want someone looking at the BSDCan schedule to say “Wow, I HAVE to go to this!” But “be interesting” isn’t exactly useful advice. Some of this might be.

1) Present an topic that hasn’t been covered before. The previous proceedings are all available online. Some topics come up every year. Don’t submit those unless you have a fresh new take on them.

2) Tell us why YOU should be the one to present your topic. We frequently get multiple similar proposals, and have to choose between them based on who we think might do the best job. Don’t assume that everyone on the committee knows who you are. One of us probably knows that you run weaselBSD on twelve thousand workstations, but they probably won’t share than information with everyone else. (Paper selection is done via web form, which isn’t as conducive to back-and-forth discussion as in-person or mailing list discussions.) More than once I’ve had someone come up to me at a conference and say “Why did you have so-and-so report on X? I have forty times as much X as the speaker you chose!” My answer is, “You didn’t TELL US that you had so much.” I am not telepathic.

3) We get many FreeBSD proposals. I want more Net/Open/Dfly proposals, as well as for projects and devices based on them. Non-F/N/O/D BSD papers have much smaller audiences, so a paper on them has to be very interesting. Again, butts in seats.

4) Include “lessons learned.” Anybody can read a man page. If you were involved in recovering from the recent FreeBSD security incident, you should submit a paper about it. If you deployed OpenBSD in a nuclear submarine and learned about the importance of the little radiation-detection badge, you should submit a paper. “Lessons learned” implies that you made mistakes, and can upgrade your talk from the merely technical to a bit of a story. Don’t be afraid to embarrass yourself, we’ve all been there. Sharing mistakes helps connect your presentation to the audience.

5) Tell us what the audience will get out of the talk — e.g., if it’s a sysadmin talk: “Attendees will be able to configure weaselBSD’s anti-ferret features and understand how they integrate with email and Web.”

6) Spelling and grammar are not pimple cream, to be dabbed on where needed. If you’re going to indulge in floccinaucinihilipilification, you’d better be able to spell it.

7) Speaking of floccinaucinihilipilification, you’re better off saying why your stuff is awesome as opposed to running down someone else’s work. I know many reasons why computers suck. Learning something to make my job suck less is good.

That’s what I look for, at least. Others have their own opinions. And if you think a little bit, you can make your proposal answer all of these. Unless you’re running weaselBSD, that is…

Now: submit!

OpenBSD ruleset tracing

As Henning reviews the Absolute OpenBSD manuscript, he’s pointed out items that I’ve missed. Some of these are only documented in man pages, while others don’t seem to really be documented anywhere except in the source code. Here’s an interesting tidbit he pointed out that I haven’t seen anywhere other than Henning’s email. (Having said this in public, I’ll now find all sorts of examples that I missed, such as Henning’s slides from EuroBSDcon 2010.)

You can log specific connections to separate log devices, for simpler debugging. You need the log rule near the beginning of your ruleset, but it doesn’t have an effect on whether a packet is passed or blocked, or how it’s translated. Here’s a pf.conf line to tell PF to watch for connections from 192.0.2.226 to 203.0.113.34.

match log (matches) from 192.0.2.226 to 203.0.113.34

Now watch the log interface, searching for the destination address:

# tcpdump -n -e -ttt -i pflog0 ip host 203.0.113.34
tcpdump: WARNING: snaplen raised from 116 to 160
tcpdump: listening on pflog0, link-type PFLOG
Dec 17 17:40:25.071592 rule 2/(match) block out on fxp0: 192.0.2.226.14033 > 203.0.113.34.822: S 2830367545:2830367545(0) win 16384 (DF)
Dec 17 17:40:25.071600 rule 5/(match) pass out on fxp0: 192.0.2.226.14033 > 203.0.113.34.822: S 2830367545:2830367545(0) win 16384 (DF)
Dec 17 17:40:25.071604 rule 5/(match) pass out on fxp0: 192.0.2.226.14033 > 203.0.113.34.822: S 2830367545:2830367545(0) win 16384 (DF)
^C

You get the number of every rule that matches the packet. If you use NAT, it prints original and destination addresses. With my wussy PF rules, this isn’t terribly exciting. With more complicated rules, however, it makes debugging rulesets much easier.

This is the sort of useful information you’ll get if you preorder the new Absolute OpenBSD. (See how I subtly slipped that in? I’m getting good at this marketing stuff.)

Absolute OpenBSD pre-orders now available

No Starch Press now has pre-orders for new Absolute OpenBSD. Order direct from the publisher, and get both the ebook and the paper for one price. If you use the coupon code ILUVMICHAEL you’ll get a discount, and I get a commission on the sale. (Bolded 20130207 because more than one person has said they missed that line.) If you use another coupon code, I still get paid, but not as much. I’m not deeply concerned which way you buy it, so long as you buy it.

Here’s the cover of the new edition. It incorporates art from the first edition, plus a new background.AO2e Cover

On a vaguely related note, I recently saw a link to my blog from a Chinese Unix users message board. Curious, I asked Google Translate what it said. It’s a discussion of the new book, which is awesome. Slightly worrying, though, is that in the translation they repeatedly refer to me as “Great God Lucas” or some variant thereof. I’m hoping that this is an artifact of translation, or some cultural thing I was previously unaware of.

Otherwise, it would seem that I have a cult of worshippers in China, and that I must learn Chinese in order to issue my commands.

Even more tangentially, links within a translated page take you to a translated version of that page. That’s pretty cool.

1st draft of Absolute OpenBSD, 2nd Ed. complete

Last night, I finished the first draft of the new edition of Absolute OpenBSD.

This is the longest book I’ve ever written (23 chapters). It’s taken longer than any other nonfiction book (3 years). Now that a first draft exists, I can state with some confidence that the book will be out about next spring-ish.

As a first draft exists, if I get trampled by a rabid caribou between now and then, the book will still come out.

This weekend is the first time in years that I will have had no work to do on the book. (Unless Henning sends me corrections on the few chapters he has left.) I plan to gaze blankly into space for several hours.

Absolute OpenBSD 2nd Edition status, 15 November 2012

Chapters 1-22 are written. Only chapter 23 remains.

The first 23 chapters are either in preliminary tech review (Henning Brauer), editing (No Starch Press), technical review (Peter Hansteen), or copyediting (No Starch Press). And every time any one of those folks are done, the chapter comes back to me for rewrites. Which is as it should be, of course… unlike some publishers, NSP gives me every chance to improve the book, as opposed to having some unpaid intern with a degree in medieval lit “fix” the text.

One chapter to go. Back to writing…