Well, just when I try to sell the organization on the importance and necesity to buy into security....bam! Well, looks like a possible root kit scenario (for how long - I don't know). I want to make absolutely sure I'm taking the right steps on this...just started to help out in the admin area so I'd like to get this right....and everyone on this forum has been fantastic in the past. Okay, so, the server is running fine (right)...pages are serving, the db is up, users aren't complaing. Things look suspect when sendmail breaks. I do not know if this was the result of a root kit, but it got a couple of us looking around. We came to the conclusion of a root kit installation after doing the following diagnostics. First, we ran chkrootkit and found some INFECTED commands and a possible LKM Trojan. Second, we found some suspect commands in one of the accounts. Three, ran an RPM verify and found many checksums off. Fourth, I looked for suspect activity on the machine with netstat-a, but couldn't find anything. Besides, netstat appears to be infrected! Now, my question is could there ever be the instance of a false positive with any of these checks? Chkrootkit? RPM Verify? What, if any, additional diagnostics would you perform? How about forensics for tracking the issue? Are there any techniques to finding how long they have been on for? Lastly, I want to backup the system (data is intact), and reinstall from scratch. However, this may be a tough sell as it appears to be the only thing to do in this scenario (assumming it is that scenario). I'm doing some reading around and this is what I've come up with (chkrootkit, RPM verify, users, services). Any tips for this scenario from the Computing.net community are appreciated very much. you guys really know your stuff :) Many thanks in advance.

Don't suppose you had tripwire installed?

Hey 3Dave, No, it's not actually my box and Tripwire was not running. Unfortunate in this case.... thanks

"could there ever be the instance of a false positive with any of these checks?" Yes, but altered binaries are probably not an accident (i.e. they shouldn't be news to you). You should have logs of the changes made to binaries. To reinforce the point, you should know that the binaries are altered or the record should inform you, as opposed to discovering. "Things look suspect when sendmail breaks" Or is exploited. Have you looked into that? The log may have omissions. Check the latest sendmail exploits. "we found some suspect commands in one of the accounts" What made them suspect? "a possible LKM Trojan" If there is a malicious LKM, the system must be shutdown or altered. You cannot trust the system if you cannot trust the kernel. "What, if any, additional diagnostics would you perform?" You can look at the entire system, but any part of it could be lying to you. Check all files with root-level (SUID) priveleges. Especially binaries. You should really have a fairly complete list of these. "How about forensics for tracking the issue?" Make a bit stream backup and copy the backup. Create a simulated configuration from 6 months ago, with updated patches, and compare configs, binaries, and other things. Consult the logs for omissions or alterations (you have to know how to read them and what to expect). Really, proper forensics requires a small obsession. Unless there is big money involved, just repair and document everything. "However, [backing up and reinstalling] may be a tough sell as it appears to be the only thing to do in this scenario" I do not understand that statement. What is hard about there only being one possible option? You either have to read through the partition tables, the filesystem, the configurations, the accounts, and then the binaries, or you can reinstall and patch. On the other hand, do you get overtime? Depending on how much money is involved with this, you should backup the system. I doubt there is too much money involved or you wouldn't be on this forum. Backup all the suspicious things, logs, and possibly changed binaries. The sendmail clutter as well. If possible, backup the entire system. If it is worth your time (again, probably not that level of money involved) make two backups, one for legal reasons, and one for investigation. Fix your primary system, unless you are brave and have left this system open for monitoring (not recommended, but great for geek story-telling). Perform the rest of your investigation via the backup. Lock down the production machine and install Tripwire. "Are there any techniques to finding how long they have been on for?" Sure, but the hacking doesn't simply happen in one sweep. Changes were made over time. Your logs are your best friend - if you back them up religiously on another machine. Still, there is sometimes nothing more alerting than a log file which is missing a few things. In short, proceed with a backup according to your means and reinstall the system. Use Bastille Linux, change root to some other name, install Tripwire, etc. If possible, change as many defaults as possible. Consider lcap.

It's probably getting a bit long in the tooth now but TrinityOS makes interesting reading too (http://freshmeat.net/projects/trinityos/)