Diagnosing “+Limiting icmp unreach response from…” with tcpdump

Anyone who has run a FreeBSD server for any length of time has seen these messages in their daily security emails. (You do read those, right?)

+Limiting icmp unreach response from 296 to 200 packets/sec
+Limiting icmp unreach response from 337 to 200 packets/sec
+Limiting icmp unreach response from 318 to 200 packets/sec
+Limiting icmp unreach response from 535 to 200 packets/sec
+Limiting icmp unreach response from 332 to 200 packets/sec
+Limiting icmp unreach response from 328 to 200 packets/sec

Way back in the Bronze Age, I learned that this mean “someone is port scanning.” The usual advice is to disable these messages by setting the sysctl net.inet.icmp.icmplim to 0. This silences the messages. I’m guilty of giving that advice myself.

What it really means is that something is sending your server UDP packets on a port that isn’t open. This could be a port scanner. It could also be a host legitimately trying to reach your host for a service it thinks you provide, or a service your host should be providing but isn’t.

I could go to my netflow collector and run a few commands to track down where these packets are coming from. In this case, the problem host is my netflow collector. I’m somewhat leery of using a tool to diagnose itself. An initial check shows that everything on the collector is running, so let’s see if it’s still happening with tcpdump.

I could run tcpdump -i em0 icmp and see all the ICMP traffic, but that’s inelegant. I don’t want to miss the traffic I’m looking for amidst a torrent of ICMP. And why have my brain filter traffic when ICMP will do it for me?

The first step is to identify exactly what we’re looking for. ICMP isn’t a monolithic protocol. Where TCP and UDP have ports, ICMP has types and codes. You can find a friendly list of types and codes here, or my readers can look in my Network Flow Analysis.

ICMP’s “port unreachable” message is type 3, code 3. Unlike TCP ports, the type and code are separate fields. Type 3 is “destination unreachable,” while the code indicates exactly what is unreachable — the port, the network, whatever. Type is ICMP field 0, while code is ICMP field 1. Tcpdump lets you filter on these just like the more familiar port numbers. Enclose more complicated filter expressions in quotes.

# tcpdump -ni em0 "icmp[0]=3 and icmp[1]=3"
10:01:03.287063 IP 10.250.250.10 > 192.0.2.214: ICMP 10.250.250.10 udp port 11022 unreachable, length 36
10:01:03.331388 IP 10.250.250.10 > 192.0.2.214: ICMP 10.250.250.10 udp port 11022 unreachable, length 36
10:01:03.356052 IP 10.250.250.10 > 192.0.2.214: ICMP 10.250.250.10 udp port 11022 unreachable, length 36
10:01:03.378256 IP 10.250.250.10 > 192.0.2.214: ICMP 10.250.250.10 udp port 11022 unreachable, length 36
10:01:03.411046 IP 10.250.250.10 > 192.0.2.214: ICMP 10.250.250.10 udp port 11022 unreachable, length 36
10:01:03.437458 IP 10.250.250.10 > 192.0.2.214: ICMP 10.250.250.10 udp port 11022 unreachable, length 36
10:01:03.457858 IP 10.250.250.10 > 192.0.2.214: ICMP 10.250.250.10 udp port 11022 unreachable, length 36

The host 192.0.2.214 is constantly trying to reach my collector on port 11022. 192.0.2.214 is my busiest border router.

That’s a router. This is a netflow collector. Maybe it’s netflow traffic? Let’s see.

# tcpdump -ni em0 -T cnfp ip host 192.0.2.214 and udp port 11022
192.0.2.214.11022 > 10.250.250.10.11022: NetFlow v5, 1897575.270 uptime, 1363184870.488773000, #1285199613, 30 recs
started 1897571.570, last 1897571.570
...

Yep. Either my router or my collector is misconfigured. And my monitoring system is misconfigured, because it should have caught that the collector process isn’t running. Or I should have noticed that I wasn’t actually getting any flow files from the collector running on another port.

Now to go back in time, find that young punk who wrote Absolute BSD, and whup his butt.