[olug] More forensic analysis

[Jeff Hinrichs]

Adam,

Thanks for sharing and the nice write up.  It is amazing how useful some
tools can be, i.e. netstat.  The other nice thing about netstat is that
it is available on just about every platform including *nix and w32.
The command line options sometimes have slight variations but netstat
-an is always a winner it seems.

-Jeff
-----Original Message-----
From: olug-bounces at olug.org [mailto:olug-bounces at olug.org] On Behalf Of
Adam Haeder
Sent: Friday, July 08, 2005 1:46 PM
To: olug at olug.org
Subject: [olug] More forensic analysis

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Another forensic analysis story for everyone. Not too complicated, but a
good example of gleaning information from your linux box.

I co-manage (part time) a small network with about 20 users. They have a
low-end (pentium something) pc running Fedora Core 3 as their
gateway/NAT box. It
also runs squid and dansguardian for filtering purposes.

I've only been involved with this box for a few months, but it runs fine
and doesn't see very much load. Got a call this morning from a user
saying
that internet requests were timing out and when the requests did go
through, it was pretty slow. Logged into the box and saw the load was
around 1.4, not huge, but a little out of the ordinary for this box. Big
Brother showed that the load had been high for about an hour. 'top'
showed
that dansguardian was the offender, using about 90% of the cpu.

I was im'ing the other admin, who is the one who set the box up. He
suggested I restart dansguardian and squid, which I did. No change; they
went back up to high cpu usage. I did a 'netstat -anp' to see what
active
connections there were to the box, and saw literally hundreds of
connections to tcp port 8080. For those of you not familiar with
netstat,
here's an example line:

tcp        0      0 1.1.1.1:8080        5.5.5.5:2841      TIME_WAIT
- -

(ips changed to protect the innocent).

In this case, the system 5.5.5.5 has a connection to tcp port 8080 on
1.1.1.1 (the server I'm working on).

Now, I'm not that familiar with dans guardian, so I at first assumed it
was a lot of internal users making web requests. The number was a little
high, though. Plus I took a couple of the IPs that were connected and
ran
them through arin.net and found out they were from places like Amsterdam
and an ISP in Canada. Not likely places to be hosting websites that
those
users would visit. Plus, the port on the other end would probably be 80
(or maybe 443) not the random high numbered port I was seeing. This
looked
more like a connection originating from there and connecting directly to
tcp/8080 on my box.

So a little more netstat showed me this:
# netstat -anp | grep LISTEN | grep 8080
tcp        0      0 0.0.0.0:8080        0.0.0.0:*               LISTEN
6294/dansguardian

And here was the problem. Dans guardian was listening by default on ALL
interfaces, not just the internal 192.168.1.1 address. And since there
was
no iptables script blocking access to this port, we had ourselves an
open
proxy server.

Now, like I said, I just started helping out with this system a few
months
ago, and had pretty much left the squid and dans guardian stuff to the
other admin. So I was surprised that if we've had an open proxy for this
long, why all of a sudden are we seeing huge traffic on it? A quick
google
search on the real static IP of the box showed that a post had been made
last night to the newsgroup alt.sex.passwords with a very long list of
open proxy servers. Sure enough, that IP was on there.

So I told the other admin what I had found and he changed the dans
guardian settings to have it only listen on the internal port. Now it
looks like this:

# netstat -anp | grep LISTEN | grep 8080
tcp        0      0 192.168.1.1:8080        0.0.0.0:*
LISTEN
6294/dansguardian

We also added an iptables rule to firewall off the port (which of course
should have been done from day 1).

What I learned from this:
- - I need to do better system audits when I get an inherited system
- - netstat, as always, is your friend. Learn how to use him. The 3 most
underappreciated but most useful diagnostic tools on linux are netstat,
lsof and strace.

Hope this helps someone out in their future diagnostic endeavors.

- --
Adam Haeder
Vice President of Information Technology
AIM Institute
adamh at omaha.org
(402) 345-5025 x115
PGP Public key: http://www.haederfamily.org/pgp.html
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFCzsn/bHC3IXlHqBQRAjAlAJ4pZ+mbeKk66BEvuJFp7ZMjIHqqHwCglWVy
mU44mzUNRZwLkdC935YYXq8=
=eO5w
-----END PGP SIGNATURE-----