Blog – Page 85 – Michael W Lucas

January 11, 2011

identifying probable intrusion vectors with flow data

Shortly after Absolute FreeBSD came out, I worked with gpart(8) and thought “I should have put this in the book.” Just after Cisco Routers for the Desperate went to the printer, I worked with tracking gateway availability and said “Drat! This should have gone into the book!” This is a recurring motif in my life.

Now that Network Flow Analysis is out, I should have marked calendar space for “interesting flow analysis opportunity.” If you want to know the details behind all of this, look in the book or in the flow-tools documentation.

Someone recently penetrated a dev server I help support. I want to learn how they got access, using flow data. I have no idea if this is realistic, but let’s go for it. I previously made a reasonable guess about the date the host was compromised, so I know the time window to examine. I’ll attack the problem by identifying “known good” traffic, removing it from the data, and examining what remains. (This might not be the best method, but I know that a couple security and intrusion response folks read this blog, and one in particular won’t hesitate to tell me I’m fubar, so check for comments.)

First, let’s see the traffic this host sends and receives.

# flow-cat 2010-11-09/ft* | flow-nfilter -F ip-addr -v ADDR=189.22.36.165 | flow-print | less

srcIP dstIP prot srcPort dstPort octets packets 189.22.36.165 194.28.157.50 6 7781 80 40 1 194.28.157.50 189.22.36.165 6 80 7781 40 1 189.22.36.165 194.28.157.50 6 9008 80 40 1 189.22.36.165 194.28.157.50 6 9008 80 40 1 194.28.157.50 189.22.36.165 6 80 9008 80 2 189.22.36.165 194.28.157.50 6 6625 80 80 2 194.28.157.50 189.22.36.165 6 80 6625 80 2 189.22.36.165 82.135.96.18 6 445 59423 80 2 82.135.96.18 189.22.36.165 6 59423 445 96 2 189.22.36.165 72.167.161.47 6 80 51428 40 1 72.167.161.47 189.22.36.165 6 51404 21 84 2 ...

This machine is an Ubuntu box. It regularly contacts random Internet sites to check for updates. The developer also browses the Web from it. If I’m to have any luck, I must exclude Web browsing traffic from this host. (To the best of my knowledge, there is not yet a Web site that will automatically root any Unix-like system. I might be wrong.) I normally configure most filtering on the command line, but this is complicated enough that I need to write an actual filter for it.

filter-primitive port80 type ip-port permit 80

filter-primitive victim type ip-address permit 189.22.36.165

filter-definition victim-browsing invert match ip-source-address victim match ip-destination-port port80 or match ip-destination-address victim match ip-source-port port80

We match all traffic from the victim machine to port 80, and from port 80 to the victim machine, then invert the filter to exclude everything that matches. Add this filter to the command line and we get:

srcIP            dstIP            prot  srcPort  dstPort  octets      packets
189.22.36.165    82.135.96.18     6     445      59423    80          2
82.135.96.18     189.22.36.165    6     59423    445      96          2
189.22.36.165    72.167.161.47    6     80       51428    40          1
72.167.161.47    189.22.36.165    6     51404    21       84          2
72.167.161.47    189.22.36.165    6     49768    21       296         6
189.22.36.165    72.167.161.47    6     21       49768    262         3
72.167.161.47    189.22.36.165    6     51428    80       40          1
...

Some interesting things here. This machine shouldn’t be running a SMB server, but the first two flows show that someone connected to us on port 445, we answered, and we sent a bunch of data. The developer owner probably installed Samba as a dependency of something else she installed, and never even noticed. Nobody on the outside world should be talking to this machine’s Web site, but it’s not that surprising that someone did. There’s a small FTP query next; I suspect it’s one of the innumerable FTP scanners.

There’s still 1,690 lines of this stuff; far too much to assess by eye. Let’s trim it down by assuming this is the most common sort of intrusion.

Generally, an intruder attacks a service on a machine. He would then send the code for the exploit or IRC bouncer to the machine through that service. Let’s make the (uncertain and unreliable) assumption that one or the other of these is larger than 1 packet. Most DNS transactions, pings, etc, are 1 packet, so by looking for flows larger than 1 packet we exclude this innocuous traffic. The following primitive and filter only passes flows larger than 1 packet.

filter-primitive gt1packet type counter permit gt 1

filter-definition gt1packet match packets gt1packet

Now add |flow-nfilter -F gt1packet to the command line and see what remains. The following immediately stands out:

...
189.22.36.165    79.115.103.225   6     22       4382     3703        19
189.22.36.165    79.115.103.225   6     22       4383     3095        11
189.22.36.165    79.115.103.225   6     6667     4384     120         3
189.22.36.165    79.115.103.225   6     6667     4385     120         3
...

The first port 6667 connections are to a host 79.115.103.225, a Romanian system. Let’s strip out all of the previous filters and see what traffic these two hosts have exchanged. There’s a lot of SSH traffic, more than we see from the usual brute-force guesser.

# flow-cat 2010-11-09/ft* | flow-nfilter -F ip-addr -v ADDR=189.22.36.165 | \
   flow-nfilter -F ip-addr -v ADDR=79.115.103.225  | flow-print | less
srcIP            dstIP            prot  srcPort  dstPort  octets      packets
79.115.103.225   189.22.36.165    6     4381     22       371         6
189.22.36.165    79.115.103.225   6     22       4381     394         7
79.115.103.225   189.22.36.165    6     4383     22       1984        14
189.22.36.165    79.115.103.225   6     22       4382     3703        19
189.22.36.165    79.115.103.225   6     22       4383     3095        11
189.22.36.165    79.115.103.225   6     6667     4384     120         3
189.22.36.165    79.115.103.225   6     6667     4385     120         3
79.115.103.225   189.22.36.165    6     4384     6667     192         3
79.115.103.225   189.22.36.165    6     4382     22       11804       118
79.115.103.225   189.22.36.165    6     4385     6667     192         3
189.22.36.165    79.115.103.225   6     22       4382     12688       103
79.115.103.225   189.22.36.165    6     4382     22       1664        19
79.115.103.225   189.22.36.165    6     4382     22       5564        64
189.22.36.165    79.115.103.225   6     22       4382     9708        50
79.115.103.225   189.22.36.165    6     4382     22       14956       169
189.22.36.165    79.115.103.225   6     22       4382     16060       129
79.115.103.225   189.22.36.165    6     4382     22       1040        12
189.22.36.165    79.115.103.225   6     22       4382     928         8
189.22.36.165    79.115.103.225   6     8888     4470     120         3
79.115.103.225   189.22.36.165    6     4470     8888     192         3
79.115.103.225   189.22.36.165    6     4382     22       4316        49
189.22.36.165    79.115.103.225   6     22       4382     11344       42
79.115.103.225   189.22.36.165    6     4382     22       1924        23
189.22.36.165    79.115.103.225   6     22       4382     8800        20
...

Using flow-print -f 5, I can view the timestamps and verify that the IRC activity started shortly after the SSH activity started using larger amounts of bandwidth.

Can I be certain that 79.115.103.225 is my attacker? No. Is this activity suspicious? Absolutely. I can examine the hacked machine, or a disk image thereof, and identify the account used to penetrate the machine.

This is not proof, but it’s a place to start. In assessing the rest of the data, I can now exclude this host. This will further reduce the pool of data I am assessing.

While I can’t use this as grounds for flying to Romania with body armor, a machine gun, and a machete, I can realistically act on this information. I can report the activity to the IP address owner. I can check my network for other connections from this host, and verify the integrity of any machines it’s connected to. I can use this a a part of my business case to firewall off this part of the network. It will support my argument to forbid passwords for SSH connections on dev machines.

In retrospect, I could have made other assumptions that might have let me find this more quickly, e.g., I could have investigated the first hosts contacted on the questionable ports. But every puzzle is easy once you’ve solved it. After this, I’d have to say that backtracking intrusion vectors through flow data is very practical, even when you don’t have much experience.

January 10, 2011January 10, 2011

dating an intrusion with flow data

One of my Ubuntu dev boxes was broken into. While the box isn’t vital, I’ll still need to reinstall an operating system and set it back up for the developer. I want to know where the attack came from and what the intruder did. I cannot trust the logs on the system, but I can trust the flow data from our upstream router.

I’ve changed my IP addresses, but remote addresses are left unchanged. Here I examine my flow data from 1 January 2011, and remove IP addresses I expect to contact this machine.

# flow-cat ft* | flow-nfilter -F ip-addr -v ADDR=189.22.36.165 | flow-print | grep -v mgmt.ip.ad.dr | grep -v dev.ip.add.dr | less srcIP dstIP prot srcPort dstPort octets packets 189.22.36.165 208.83.20.130 6 60702 6667 198 3 189.22.36.165 208.83.20.130 6 60703 6667 196 3 208.83.20.130 189.22.36.165 6 6667 60702 152 3 208.83.20.130 189.22.36.165 6 6667 60703 152 3 ...

So, what do I learn here? This system was compromised on or before New Years’ Day. 208.83.20.130 is an IRC server, and 6667 is an IRC port. Someone is using my system to play IRC games. Bastards. Other checks show that the intruders are also using port 7000.

I don’t know exactly when the system was compromised. Fortunately, I have my old flow records. I go back and check the first of each previous month, narrowing down the time window. 1 December looks like 1 January, but 1 November looks different:

srcIP dstIP prot srcPort dstPort octets packets 189.22.36.165 189.22.37.222 17 123 123 76 1 189.22.36.222 189.22.36.165 17 123 123 76 1 206.80.36.88 189.22.36.165 17 65015 5060 368 1 189.22.36.165 206.80.36.88 1 0 771 396 1 129.82.138.38 189.22.36.165 1 0 2048 28 1 ...

The port 123 UDP traffic is NTP. And someone poked at me with a SIP client, but we didn’t answer. This is about what I’d expect to see on a machine sitting naked on the Internet.

Next, search to narrow down the time window. When I find the first day the IRC server traffic appears, I know when to start looking for the actual intrusion activity. When was the first day that port 6667 and 7000 traffic appeared? It was present on 1 December, but not 1 November. Check November 15: present, November 7: present, etc, etc. Eventually, I see the traffic is present on 9 November, but not on 8 November.

The intrusion happened on or before 9 November 2011.

Next I will examine the traffic for 8 and 9 November and see if I can determine where the intruders came from and their attack vector. I haven’t done that analysis yet, so who knows what I’ll find, if anything, but I’ll post on my efforts one way or another.

UPDATE: Oh, right, I’m an author. While I shouldn’t blatantly pimp myself out here, when I do an on-book-topic post, I should at least say “Hey, if you want to do this too, you can learn how by reading my newest book.” Sheesh. Being non-commercial is one thing, being actively daft is another.

January 5, 2011January 19, 2011

mod_security rule upgrades and logging

I recently installed mod_security2 on my personal Web server to block out the most annoying referral spam. It blocked the worst offenders. Then I found that mod_security also included the ability to block all access from sites in a DNS-based RBL. This would further reduce my comment and referral spam problems, at the cost of making the site slightly slower. There are also rules to block SQL injection attacks and other known attack vectors. If you’re trying to read this from a machine on DNS blacklists, stop reading and go get yourself off the blacklist.

The RBL rules aren’t in the base FreeBSD package. They’re in the newer mod_security2 ruleset, available from mod_security’s Sourceforge download page. Get the newest file.

Move your existing rules to a safe place, and put the new rules where the old rules were. Do not delete your old rules, you’ll want them for reference. In my case, the active rules directory is /usr/local/etc/apache22/Includes/mod_security2. I moved the existing directory to /usr/local/etc/apache22/Includes/old-mod_security2, created a new mod_security2 directory, and unzipped the rules it there.

The rules directory contains an example rule file, modsecurity_crs_10_config.conf.example. Apache will read any file that ends in .conf as a config file, so copy (not move) that example to modsecurity_crs_10_config.conf. Edit that file to include changes from the original setup, e.g.:

SecRuleEngine On SecDataDir /var/run/modsecurity

Copy your referer.conf referrer blacklist into the new rules directory. Then reload Apache. If Apache won’t restart, read the error messages and correct them.

Now that you have the base rules upgraded, you can add rules from the optional_rules directory. I specifically want the comment spam blocking, so I copied modsecurity_crs_42_comment_spam.conf to the main directory and reloaded Apache.

Then use wget to test my work, using one of the less offensive referral spam sites as a referrer. (I’ve changed the name of the site to avoid giving them any more links.)

avarice/tmp$ wget http://www.michaelwlucas.com/ --referer=http://www.fishingscum.com

–2011-01-04 17:04:06– http://www.michaelwlucas.com/
Resolving www.michaelwlucas.com (www.michaelwlucas.com)… 198.22.63.8
Connecting to www.michaelwlucas.com (www.michaelwlucas.com)|198.22.63.8|:80… connected.
HTTP request sent, awaiting response… 500 Internal Server Error
2011-01-04 17:04:06 ERROR 500: Internal Server Error.

That’s what I want.

As I had to take the time to upgrade, I wanted to also get a log of what hits I was blocking. This only took adding two lines to the configuration:

SecDebugLogLevel 1 SecDebugLog /var/log/modsecurity.log

My wget request generated this log entry:

[04/Jan/2011:17:04:06 --0500] [www.michaelwlucas.com/sid#801948060][rid#801aa20a0][/][1] Access denied with code 500 (phase 2). Pattern match "fishingscum" at REQUEST_HEADERS:Referer. [file "/usr/local/etc/apache22/Includes/mod_security2/referer.conf"] [line "35"]

Setting SecDebugLogLevel to 2 gave me details on how mod_security2 processed its logs. That will be useful if I ever have to write my own mod_security2 rules. I suspect that if I have to do that, though, I’m solving the wrong problem.

One interesting thing I saw here was how the log statement in mod_security2 rules is applied. If you use the log keyword in a rule, a log message appears in the standard Apache access and error logs as well as the mod_security2. If you do not use the log statement, a message appears in the modsecurity log but not in the Apache logs. An anti-referral-spam rule should look like this:

SecRule REQUEST_HEADERS:REFERER "ezinearticles" deny,status:500

24 hours later, WordPress shows only 5 comments in my anti-spam queue. Another annoyance quashed.

UPDATE: More here.

January 3, 2011

next tech book outline

My big project for the holidays was completing an outline for the next tech book. I’m glad to say that the outline is done. I can’t yet give the title, but I can say:

It’s huge: 29 chapters in a 25-page outline.
I have a tech editor, a respected figure in the relevant community. He’s currently reviewing the outline.
No Starch Press wants it.
I expect to spend a year writing it, so I would expect a release in early 2012.

The book’s length is a concern. I want to write books small enough for me to hold comfortably in the bathtub. I’m a big guy, but a 29-chapter tech book pushes that limit. I might trim some content, or cover some parts in less detail, to meet that goal.

I don’t announce book titles far in advance, due to problems that’s caused me in the past. (Some day I’ll write up that story, but not today.) I expect that I’ll be far enough along in a few months to announce the title, so: I’ll announce it during my presentation at BSDCan 2011. (And now that I’ve made a public commitment to that date, I’ll have to get cracking!)

December 30, 2010

I have achieved pole position

I sold my short story Wednesday’s Seagulls to short-story.me a few months ago. They released that story as part of their second “best of” anthology collection in November. I’ve wanted to check it out, but the holidays and my efforts to outline another nonfiction book interfered.

The order of stories within a volume takes more thought than most people realize. Most of the big-name editors have their own occult ordering methods, but there’s a few general rules. The first story in the anthology is the story that, in the editor’s opinion, is most likely to hook the reader and compel them to read further. The last story is the one that, in the editor’s opinion, is most likely to leave the reader with the a good impression of the anthology. While I’m proud of my previous antho sales, my work therein is buried in the comfortable middle.

Wednesday’s Seagulls is the first story in this anthology. I have been awarded pole position. Merry Christmas to me! You can get the anthology in print and on Kindle.

I really must learn to write faster. Maybe if I give up eating and sleeping…

December 23, 2010

Blowing up the Holidays

As a special Christmas present to myself, I’m solving a problem and making a positive improvement to my environment. Using Perl and gnuplot. I’m not going to share the actual code, for two reasons: one, it’s very specific to an in-house problem, and two, I use a programming technique I call “iterative petulance.”

So instead, here’s something for the holidays: land mines.

Land mines are bad. Land mines that we’ve left lying around are really bad. Clearing abandoned land mines is hard, dangerous, and expensive work. My new favorite charity clears these landmines inexpensively and safely… using rats. No, you don’t herd rats across minefields. Hero Rats are trained to smell explosives, and are too small to set off the mines.

Most of us have too much stuff. Why not adopt a rat for someone instead of giving them yet more stuff?

Hopefully, I’ll be doing something technically interesting early next year. In the meantime, happy holidays!

December 15, 2010

The OpenBSD IPSec kerfuffle

By now you’ve probably heard of the allegations Theo forwarded to the OpenBSD-tech mailing list about the FBI introducing back doors in early versions of the OpenBSD IPSec code. I’d like to offer my opinion, in the spirit of the Christmas season:

“Bah, humbug!”

It’s possible, but unlikely. Like me winning the lottery is unlikely. I’d need to buy a ticket, and that isn’t going to happen any time soon.

The OpenBSD group examines every line of code that goes into their tree. Any obvious back door would be caught. Any subtle back door would be fragile — so subtle that it probably wouldn’t survive the intervening ten years of code churn and IPSec improvements. Maybe someone has an appliance based on, say, OpenBSD 2.8 or 3.2, which could have contained the back door. If true, we need to know about it. But those users need to upgrade anyway.

And the FBI? Nope, don’t believe it. Ten years ago, the FBI was having lots of trouble understanding the Internet. The NSA, maybe.

Bugs? Sure, there’s probably bugs. I expect we’ll find some, now that many eyes have turned to the code. Exploitable bugs? Maybe. But that’s not the same as a back door.

OpenBSD has claimed to be the best for many years. That claim motivates people to take them down. The claims have hopefully inspired many people to examine the current and historical IPSec stack. Theo and company have done nothing to discourage such audits: they’ve even offered pointers on where to look. If you’re a programmer looking to make a splash, you could do worse than to join in on auditing the code. Finding the alleged back door would make your reputation. And we can always use more IPSec hackers.

The real impact might be, as Jason Dixon points out, the cost in OpenBSD developer time. You know that some of their committers are examining the IPSec code today, trying to find potential back doors.

December 13, 2010

designing a tech book

So, you’ve figured out some incremental advance in your own education that you think would make a good book. Now you grab your keyboard, open a text file, and start typing.

Not so fast.

Just as a large programming project has a specification, a book has a design. You can start churning out text just as you might churn out code, but eventually you’ll have to stop and think about what you’re doing. Time spent designing your book beforehand will pay off in the actual writing of the book. Here’s how I do it; other authors have their own methods, of course, and you should do whatever works for you.

When I consider writing a book, one of the first things I do is write down what I think should be in the book, without any detail whatsoever. If I’m writing a book on a particular operating system, it should include installation instructions, guidance on the community around the operating system, the sysctl setting that bit me yesterday, upgrade instructions, and so on. Everything that I think of, both large and small, trivial and vital.

Keep this list with you as you experiment and learn. When you think of something else that should go in the book, add it to the list. No detail is too small. If your idea is strong enough and sufficiently broad to carry an entire book, you’ll eventually have 30-40 pages of “stuff.”

Take a couple hours and try to sort this list into broad categories. Move details around from category to category. See what facts they belong with. Those categories will eventually become chapters.

Now comes the annoying bit. Think about what the reader needs to know before he can understand one of these sections. A properly designed book can be read from beginning to end, each paragraph presenting the reader with new knowledge that is built on top of what he’s read before. This order of information is vital to reader comprehension.

For example, look at Absolute FreeBSD. In my original notes, I had a category called “Managing Disks.” I had another category called “Security.” The book needed to cover both topics. But what order should they go in?

To use some advanced security tools, such as jails, the reader must be able to manage system disks. Put the disk management section first.
The first thing a sysadmin needs to do is create a nonprivileged account for his routine work. User management is definitely part of security. The security section must go first.
Encrypting a disk partition is part of security… and part of disk management. You need a decent understanding of both before you can manage an encrypted partition.
You don’t have to understand security to manage software RAID, so it’s irrelevant either way.
The reader needs both before he can realistically upgrade his system.

The solution: split the disk category into Chapter 8 and Chapter 18, and split security into Chapter 7 and Chapter 9. Repeat this process for every category in your list. Some categories will be too small to make up an entire chapter, and you’ll find that you have to shuffle them into another chapter. User management could have been its own chapter, but it was small enough that I added it to the first system security chapter.

Assess each category based on what the reader needs to know before he can understand it, and place it appropriately in order. Spend some time on this; fixing ordering problems is easiest before you start writing.

Then arrange topics within categories. My first security chapter included both user management and file flags. Which one should go first? Separate your notes into further piles within each category.

Now that you have a list of chapters, and a sample of stuff in each chapter, fill out your chapters. Your long list might have said “add users” and “disable users.” Now that you have both notes side-by-side, you can see that you should probably add “remove users.” Read man pages on the topics. Fill in stuff that you should cover now that you’re being methodical.

If you have to move chunks from one category to another, that’s fine. Pick up the whole hierarchy and move it.

Eventually, your mass of points will become a design for your book. Set it aside for a few days, and come back to it with fresh eyes. Are any of the chapters far too short? Combine them. Do any early chapters require knowledge that you’ll cover in later chapters? Rearrange them. Did you miss any topics? Add them. Are any of these topics redundant, uninteresting, or otherwise bogus? Cut them.

These days I do all this rearranging using OpenOffice and bullets. That’s not ideal — I’d like to be able to expand and shrink chapters, and have checkboxes for each item so I could record when it was done. I used yank for several years, but it no longer builds and I haven’t found anything equally flexible and simple. If you have a comfortable tool, use it, but the point is, you don’t need much more than a text editor.

With careful thought, this will create a book outline that’s both useful to you and (comparatively) easy to write.

December 10, 2010

The Wikileaks/BSD connection

I was amused to discover the connection between Wikileaks and BSD.

Apparently Julian Assange hung around the BSD community up until ten years ago, and has a few entries in the NetBSD fortune files. (Search for Julian Assange in the file, or just click on the next link for the best ones.) He lived in the house where Greg Lehey grew up, although many years after Greg had moved on. Greg was interviewed for a story in the Australian news. They botched it.

If you think about it, you’d realize the connection must go deeper than that. We all know about Osama bin Lehey. Apparently the house where Greg was raised has that effect on people. I do believe that Lovecraft wrote a story about that… and it will bug me until I can remember which story that was.

December 9, 2010

writing tech books: write what you don’t know

This is the first of an irregular series on writing tech books. I got the idea from something Richard Beijtlich wrote in a review years ago. Unfortunately I cannot find the cite, and spending the day exhaustively reading my old reviews is psychologically unhealthy, but he said something along the lines of “tech authors could do worse than see how Lucas does things.” I believe that the average quality of writing in tech books is abominable. Perhaps I can pull those standards up. By the ear, if necessary.

So, where do you start to write a tech book? Start by deciding what you want to write about. One of the old cliches about writing is that you should “write what you know.” In tech writing this is not only wrong, but actively harmful.

Writing about something you already understand, in a way where you can communicate your knowledge to the uninitiated, is hard. You brain contains a lot of information, and it’s all jumbled together, interconnected. If you think your desk is messy, your brain is worse. Teasing out what you know, and how you know it, with the necessary context to explain it to someone else, is hard.

Instead, I recommend writing about something you want to know about.

When I started writing PGP & GPG, I certainly wasn’t an encryption ninja. I run BSD servers, but when I started writing the FreeBSD and OpenBSD books I didn’t possess the breadth of mastery necessary to write a book about either. I learned as I wrote the books. The books actually became structured learning — self-directed, self-designed, at my own pace, but highly effective.

As you learn the topic, take notes. Write down what what you must do to accomplish a task. Use script(1) and screenshots. Get a paper notebook, keep it by your computer, and scribble notes in it. This gives you a student’s perspective. As you learn, you’ll see how this new piece of knowledge attaches to the other knowledge in your head. Notice those mental connections as they happen. Those are things you should mention in your writing. Part of your job as an author is to help the reader make those connections inside his own head.

Note that you should choose a topic that is an incremental advance on what you already know. You need a base of knowledge to learn from. Trying to write on something that seems close to to your skill set, but isn’t truly incremental, will fail. Suppose I wanted to write a book on Perl. I write Perl, but my code is appalling. To write the book on Perl would require that I first become a programmer, then learn Perl. This is not an incremental education. If I want to write a book on, say, Tuvan throat singing, I would need considerably more lead time and a much higher budget. (Plus completely deaf neighbors.)

By writing about the next step in your education, you’ll expand your own knowledge about the topic. I might not have been a netflow expert when I started writing Network Flow Analysis, but I sure know a heck of a lot more about it than I did when I started.