Find the source of malware?

I have a server that was running an older version of lighttpd (1.4.19 on a freebsd 6.2-RELEASE (yea, old) machine) and google alerted me that it had found malware embedded on one of my server's pages. It just so happened to be our index page. I promptly removed the malware and started looking at server logs for how it got there. With no trace in any of the logs of the files being edited, I noticed that the index page's owner had been changed to www, which is the lighttpd user. I then concluded that some sort of veunerability must have existed for that software version and promptly upgraded to 1.4.26.

Now the malware is back. I have started some pretty verbose server logging with ftp, lighttpd, and all login attempts to try and see how this script is getting in. Are their any suggestions as to other approaches to take?

Your website was compromised/defaced and when this happens, it's normally very difficult to re-create all the attackers steps and the best solution is to reinstall the compromised servers. On the other hand, you need to perform some forensics in order to figure out what has possibly happened and preventing it from happening again.

Here's a list of things worth checking:

• see if there are known vulnerabilities in your webserver and your ftp server versions
• take a look at every logfiles you can, especially the webserver, the ftp server and the system ones. In the webserver logfiles, check for posts
• are there any services running that you don't need? Are they accessible from the Internet? Shut them now, check their logs and check for possible known vulnerabilities.
• run rootkit checkers. They're not infalible but can lead you in the right direction. chkrootkit and especially rkhunter are the tools for the job
• run nmap from outside of your server and check if there is something listening on any port that shouldn't be.
• if you've a rrdtool trending application (like Cacti, Munin or Ganglia) take a look the graphs and search for a possible time frame of the attack.

Also, always keep this is mind:

• shut down all the services you don't need
• backup everything you need to rebuild your server and test backups on a regular basis
• follow the least privilege principle
• have your services updated, especially regarding security updates
• don't use default credentials

Hope this helps!