dating an intrusion with flow data

One of my Ubuntu dev boxes was broken into. While the box isn’t vital, I’ll still need to reinstall an operating system and set it back up for the developer. I want to know where the attack came from and what the intruder did.  I cannot trust the logs on the system, but I can trust the flow data from our upstream router.

I’ve changed my IP addresses, but remote addresses are left unchanged. Here I examine my flow data from 1 January 2011, and remove IP addresses I expect to contact this machine.


# flow-cat ft* | flow-nfilter -F ip-addr -v ADDR=189.22.36.165 | flow-print | grep -v mgmt.ip.ad.dr | grep -v dev.ip.add.dr | less
srcIP            dstIP            prot  srcPort  dstPort  octets      packets
189.22.36.165    208.83.20.130    6     60702    6667     198         3
189.22.36.165    208.83.20.130    6     60703    6667     196         3
208.83.20.130    189.22.36.165    6     6667     60702    152         3
208.83.20.130    189.22.36.165    6     6667     60703    152         3
...

So, what do I learn here? This system was compromised on or before New Years’ Day. 208.83.20.130 is an IRC server, and 6667 is an IRC port.  Someone is using my system to play IRC games. Bastards. Other checks show that the intruders are also using port 7000.

I don’t know exactly when the system was compromised.  Fortunately, I have my old flow records.  I go back and check the first of each previous month, narrowing down the time window.  1 December looks like 1 January, but 1 November looks different:

srcIP            dstIP            prot  srcPort  dstPort  octets      packets
189.22.36.165    189.22.37.222    17    123      123      76          1
189.22.36.222    189.22.36.165    17    123      123      76          1
206.80.36.88     189.22.36.165    17    65015    5060     368         1
189.22.36.165    206.80.36.88     1     0        771      396         1
129.82.138.38    189.22.36.165    1     0        2048     28          1
...

The port 123 UDP traffic is NTP.  And someone poked at me with a SIP client, but we didn’t answer. This is about what I’d expect to see on a machine sitting naked on the Internet.

Next, search to narrow down the time window. When I find the first day the IRC server traffic appears, I know when to start looking for the actual intrusion activity. When was the first day that port 6667 and 7000 traffic appeared?  It was present on 1 December, but not 1 November.  Check November 15: present, November 7: present, etc, etc.  Eventually, I see the traffic is present on 9 November, but not on 8 November.


# cd /var/db/flows/rtr8/2010/2010-11
# flow-cat 2010-11-09/ft* | flow-nfilter -F ip-addr -v ADDR=189.22.36.165 | flow-nfilter -F not-ip-port -v PORT=80 | flow-nfilter -F not-ip-port -v PORT=53 | flow-nfilter -F not-ip-port -v PORT=123 | flow-print | grep -c 7000
947

# flow-cat 2010-11-08/ft* | flow-nfilter -F ip-addr -v ADDR=189.22.36.165 | flow-nfilter -F not-ip-port -v PORT=80 | flow-nfilter -F not-ip-port -v PORT=53 | flow-nfilter -F not-ip-port -v PORT=123 | flow-print | grep -c 7000
0

The intrusion happened on or before 9 November 2011.

Next I will examine the traffic for 8 and 9 November and see if I can determine where the intruders came from and their attack vector. I haven’t done that analysis yet, so who knows what I’ll find, if anything, but I’ll post on my efforts one way or another.

UPDATE: Oh, right, I’m an author. While I shouldn’t blatantly pimp myself out here, when I do an on-book-topic post, I should at least say “Hey, if you want to do this too, you can learn how by reading my newest book.” Sheesh. Being non-commercial is one thing, being actively daft is another.