How to find an unauthorized person's entry point to a server

We're running an in house dedicated server with 100's of sites on it. Some of these sites were very insecure and the box was not managed very well. Some one got in, messed up a bunch of stuff and is pestering us by defacing some of the sites, asking for money etc.

We made a lot of fixes but there is still some sort of backdoor through which he can get in. We found and deleted a few files that were a kind of hacker control panel but since the file was just called contactUs.php, it's impossible to know that all instances of these kinds of files were removed.

I realise this is a very broad subject and a question that may be difficult to answer but what steps would one undertake to find out how this person is getting in to the system?

It's a Fedora machine running mainly PHP sites.

Honestly, if you're running a business I wouldn't try to study the attacker, I'd just lock them out and move on. It's just not worth the time, hassle, and frustration of getting into a battle of wits with someone like this. Some ideas for getting rid of them:

• Reinstall the box, from scratch, using known-good media.
• Before you start putting any sites on it, make sure it's completely patched and up to date. If you're running an out-of-date Fedora release (one that isn't actively receiving security updates) then upgrade to the latest release (or use something with proper security support).
• Don't run PHP scripts via mod_php, because it's child's play to exploit the inevitable permissions problems that result from having to give the webserver write access to various parts of the system. Use suexec or suphp.
• Run PHP scripts as a separate user from the main "FTP account" (or equivalent), so if you have a user example (for the site example.com) then run their dynamic content in a user called example-cgi (or equivalent). Then only grant the example-cgi user write access to those parts of the filesystem tree that require them. (This requires modifying suexec's permissions checking, but it's worth it). This way the attacker can't run scripts that modify the site's PHP code, and they have to work much harder to inject an exploit.
• Seek and destroy all world-writable files on a proactive, on-going basis.
• Restrict FTP and SSH connections from known locations, rather than allowing the entire Internet to connect.
• Enforce strong passwords using something like PAM's cracklib module -- or, better still, get rid of FTP entirely and require the use of SSH keys to get into the machine.

Probably a pile of other things I can't think of at the moment, but that'll keep you going for a while.

Some ideas:

If you can get evidence of the files which he altered you could probably get a timestamp which you could correlate with some log on the server itself or an intermediate box (the firewall or so). If this guy cleans up after himself, have duplicate logs or set up remote logging on the box where he logs in - this way he cannot erase the evidence. The cross-correlation is going to be tricky to say the least, especially with your load.

Other than that, use rkhunter, chkrootkit and OSSEC and see what it turns up. Or set up a honeypot and see if he takes the bite :)