Tracing linux server hacking

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
We had an instance this weekend where our network was bought to a
grinding halt by what we believed was a problem with our network
service provider.  After 8-9 hours of our network professionals looking
into the problem, they finally diagnosed the problem with my web server
by unplugging it from the network and seeing instant improvements.

I then started diagnosing the server and was getting no response - it
too had come to a grinding halt due to the massive amounts of unknown
activity.  I was left with no choice but to reboot as I was getting no
result from any commands.

After a reboot the system came up fine and has now been operating as
per normal for the past 24 hours.  The problem I have is I don't know
what happened or where to start looking!!!  I have an operations guy
who is overseas on annual leave and have limited experience in his

Can anyone suggest where I can start or any resources I can look into?
Any advice or guidance would be greatly appreciated.

The main machine the copped the brunt of the problem was:

Linux 2.4.9-e.27smp #1 SMP Tue Aug 5 15:49:54 EDT 2003 i686 unknown

The other machine which may have also copped some activity:

Linux 2.4.18-14smp #1 SMP Wed Sep 4 12:34:47 EDT 2002 i686 i686 i386

Re: Tracing linux server hacking

Quoted text here. Click to load it

Well, yoy should start checking the logs (if any is left) to see which
process was using the most of the machine, without a real insight in
what the machine was doing you can't really be sure that it was an hack
attempt or something else. Did you updated/upgraded the machine lately?

Do you have some kind of automatic procedure in there that automatically
check the logfiles for strange activity? If not, this is a good time to
begin with.

Quoted text here. Click to load it

Also, it would be time to upgrade the kernels. And mabye something else.


I think I'd like to see a Simpsons episode start up with Bart Simpson
writing 'I will not attempt to undermine the Usenet cabal'.
--J.D. Falk

Re: Tracing linux server hacking wrote:

Quoted text here. Click to load it

The usual advice is to not boot a possibly compromised hard disk.
Save to tape for possible forsenics, better yet yank the hard disks
and replace with backups.
At least, wipe and restore from known good backups.

At least you should have save the logs from /var, save /bin, sbin and  
a copy of the kernel.

There are any number of live CD forsenics disks out there.
Such as Helix, Knoppix with forsenics suites.
Usually you start looking at logs. Sometimes a compromised
system will have the logs wiped and faked.  So part
of forsenics may be looking back at logs for anomolies
showing signs of a wipe job, showing you have been compromised
and about when. Only a newb script kiddy would fail to tamper
with logs to cover his or her tracks.

You want to know, have we been purposefully hacked?
Was it an inside or outside job?  When?
When is important as it tells you what backups are probably
compromised or not.

Live CD disks like this are also good for checking the usual
supected binaries for tampering.  Finding the true size of ls
for example.

Today's hackers are moving to hacking kernels.
Better live CD forsenics CDs are good for checking for this.
You cannot trust a possibly compromised system tools to find out.

Quoted text here. Click to load it


When I shake my killfile, I can hear them buzzing!

Cheerful Charlie

Site Timeline