identifying probable intrusion vectors with flow data

Shortly after Absolute FreeBSD came out, I worked with gpart(8) and thought “I should have put this in the book.”  Just after Cisco Routers for the Desperate went to the printer, I worked with tracking gateway availability and said “Drat!  This should have gone into the book!”  This is a recurring motif in my life.

Now that Network Flow Analysis is out, I should have marked calendar space for “interesting flow analysis opportunity.”  If you want to know the details behind all of this, look in the book or in the flow-tools documentation.

Someone recently penetrated a dev server I help support. I want to learn how they got access, using flow data.  I have no idea if this is realistic, but let’s go for it.  I previously made a reasonable guess about the date the host was compromised, so I know the time window to examine. I’ll attack the problem by identifying “known good” traffic, removing it from the data, and examining what remains. (This might not be the best method, but I know that a couple security and intrusion response folks read this blog, and one in particular won’t hesitate to tell me I’m fubar, so check for comments.)

First, let’s see the traffic this host sends and receives.

# flow-cat 2010-11-09/ft* | flow-nfilter -F ip-addr -v ADDR=189.22.36.165 | flow-print | less

srcIP            dstIP            prot  srcPort  dstPort  octets      packets
189.22.36.165    194.28.157.50    6     7781     80       40          1
194.28.157.50    189.22.36.165    6     80       7781     40          1
189.22.36.165    194.28.157.50    6     9008     80       40          1
189.22.36.165    194.28.157.50    6     9008     80       40          1
194.28.157.50    189.22.36.165    6     80       9008     80          2
189.22.36.165    194.28.157.50    6     6625     80       80          2
194.28.157.50    189.22.36.165    6     80       6625     80          2
189.22.36.165    82.135.96.18     6     445      59423    80          2
82.135.96.18     189.22.36.165    6     59423    445      96          2
189.22.36.165    72.167.161.47    6     80       51428    40          1
72.167.161.47    189.22.36.165    6     51404    21       84          2
...

This machine is an Ubuntu box.  It regularly contacts random Internet sites to check for updates.  The developer also browses the Web from it.  If I’m to have any luck, I must exclude Web browsing traffic from this host.  (To the best of my knowledge, there is not yet a Web site that will automatically root any Unix-like system.  I might be wrong.)  I normally configure most filtering on the command line, but this is complicated enough that I need to write an actual filter for it.


filter-primitive port80
type ip-port
permit 80

filter-primitive victim
type ip-address
permit 189.22.36.165

filter-definition victim-browsing
invert
match ip-source-address victim
match ip-destination-port port80
or
match ip-destination-address victim
match ip-source-port port80

We match all traffic from the victim machine to port 80, and from port 80 to the victim machine, then invert the filter to exclude everything that matches. Add this filter to the command line and we get:

srcIP            dstIP            prot  srcPort  dstPort  octets      packets
189.22.36.165    82.135.96.18     6     445      59423    80          2
82.135.96.18     189.22.36.165    6     59423    445      96          2
189.22.36.165    72.167.161.47    6     80       51428    40          1
72.167.161.47    189.22.36.165    6     51404    21       84          2
72.167.161.47    189.22.36.165    6     49768    21       296         6
189.22.36.165    72.167.161.47    6     21       49768    262         3
72.167.161.47    189.22.36.165    6     51428    80       40          1
...

Some interesting things here. This machine shouldn’t be running a SMB server, but the first two flows show that someone connected to us on port 445, we answered, and we sent a bunch of data. The developer owner probably installed Samba as a dependency of something else she installed, and never even noticed. Nobody on the outside world should be talking to this machine’s Web site, but it’s not that surprising that someone did. There’s a small FTP query next; I suspect it’s one of the innumerable FTP scanners.

There’s still 1,690 lines of this stuff; far too much to assess by eye.  Let’s trim it down by assuming this is the most common sort of intrusion.

Generally, an intruder attacks a service on a machine. He would then send the code for the exploit or IRC bouncer to the machine through that service.  Let’s make the (uncertain and unreliable) assumption that one or the other of these is larger than 1 packet.  Most DNS transactions, pings, etc, are 1 packet, so by looking for flows larger than 1 packet we exclude this innocuous traffic.  The following primitive and filter only passes flows larger than 1 packet.

filter-primitive gt1packet
type counter
permit gt 1

filter-definition gt1packet
match packets gt1packet

Now add |flow-nfilter -F gt1packet to the command line and see what remains. The following immediately stands out:

...
189.22.36.165    79.115.103.225   6     22       4382     3703        19
189.22.36.165    79.115.103.225   6     22       4383     3095        11
189.22.36.165    79.115.103.225   6     6667     4384     120         3
189.22.36.165    79.115.103.225   6     6667     4385     120         3
...

The first port 6667 connections are to a host 79.115.103.225, a Romanian system. Let’s strip out all of the previous filters and see what traffic these two hosts have exchanged. There’s a lot of SSH traffic, more than we see from the usual brute-force guesser.

# flow-cat 2010-11-09/ft* | flow-nfilter -F ip-addr -v ADDR=189.22.36.165 | \
   flow-nfilter -F ip-addr -v ADDR=79.115.103.225  | flow-print | less
srcIP            dstIP            prot  srcPort  dstPort  octets      packets
79.115.103.225   189.22.36.165    6     4381     22       371         6
189.22.36.165    79.115.103.225   6     22       4381     394         7
79.115.103.225   189.22.36.165    6     4383     22       1984        14
189.22.36.165    79.115.103.225   6     22       4382     3703        19
189.22.36.165    79.115.103.225   6     22       4383     3095        11
189.22.36.165    79.115.103.225   6     6667     4384     120         3
189.22.36.165    79.115.103.225   6     6667     4385     120         3
79.115.103.225   189.22.36.165    6     4384     6667     192         3
79.115.103.225   189.22.36.165    6     4382     22       11804       118
79.115.103.225   189.22.36.165    6     4385     6667     192         3
189.22.36.165    79.115.103.225   6     22       4382     12688       103
79.115.103.225   189.22.36.165    6     4382     22       1664        19
79.115.103.225   189.22.36.165    6     4382     22       5564        64
189.22.36.165    79.115.103.225   6     22       4382     9708        50
79.115.103.225   189.22.36.165    6     4382     22       14956       169
189.22.36.165    79.115.103.225   6     22       4382     16060       129
79.115.103.225   189.22.36.165    6     4382     22       1040        12
189.22.36.165    79.115.103.225   6     22       4382     928         8
189.22.36.165    79.115.103.225   6     8888     4470     120         3
79.115.103.225   189.22.36.165    6     4470     8888     192         3
79.115.103.225   189.22.36.165    6     4382     22       4316        49
189.22.36.165    79.115.103.225   6     22       4382     11344       42
79.115.103.225   189.22.36.165    6     4382     22       1924        23
189.22.36.165    79.115.103.225   6     22       4382     8800        20
...

Using flow-print -f 5, I can view the timestamps and verify that the IRC activity started shortly after the SSH activity started using larger amounts of bandwidth.

Can I be certain that 79.115.103.225 is my attacker? No. Is this activity suspicious? Absolutely. I can examine the hacked machine, or a disk image thereof, and identify the account used to penetrate the machine.

This is not proof, but it’s a place to start. In assessing the rest of the data, I can now exclude this host. This will further reduce the pool of data I am assessing.

While I can’t use this as grounds for flying to Romania with body armor, a machine gun, and a machete, I can realistically act on this information. I can report the activity to the IP address owner. I can check my network for other connections from this host, and verify the integrity of any machines it’s connected to. I can use this a a part of my business case to firewall off this part of the network. It will support my argument to forbid passwords for SSH connections on dev machines.

In retrospect, I could have made other assumptions that might have let me find this more quickly, e.g., I could have investigated the first hosts contacted on the questionable ports. But every puzzle is easy once you’ve solved it. After this, I’d have to say that backtracking intrusion vectors through flow data is very practical, even when you don’t have much experience.

Stalk me on social media

One Reply to “identifying probable intrusion vectors with flow data”

Comments are closed.