How to analyse logs after the site was hacked

One of our web-projects was hacked. Malefactor changed some template files in project and 1 core file of the web-framework (it's one of the famous php-frameworks). We found all corrupted files by git and reverted them. So now I need to find the weak point.

With high probability we can say, that it's not the ftp or ssh password abduction. The support specialist of hosting provider (after logs analysis) said that it was the security hole in our code.

My questions:

1) What tools should I use, to review access and error logs of Apache? (Our server distro is Debian).

2) Can you write tips of suspicious lines detection in logs? Maybe tutorials or primers of some useful regexps or techniques?

3) How to separate "normal user behavior" from suspicious in logs.

4) Is there any way to preventing attacks in Apache?

Thanks for your help.

Solution 1:

Like HopelessNOOb, I'd recomend getting some professional help with this.

When you know that a system has been compromised you can't rely on any data stored on it. Further, if the compromise was via HTTP, then there's probably not enough information in the standard logs to be able to isolate what hapenned. This is why people who are serious about webserver security will use something like mod_security to get much more detailed logs, and run a host based IDS to detect compromises.

The support specialist of hosting provider (after logs analysis) said that it was the security hole in our code

If he/she can make that claim then he/she must already know the answer to your question - either that or the claim is not substantiated and they really mean to say that they couldn't find anything wrong with their stuff.

Your objective should be to get your site back online and secure. To make a site secure, you need to plug every hole - but to compromise a site you only need to find one hole: you know you have at least one hole in your site, but you need to fix any which exist - which means you need to take a very different approach to your site security. Even of you can identify and fix the vulnerability which was exploited this time, the evidence shows that you don't have the processes and skills in place to have any sort of confidence that this was an isolated incident.

Solution 2:

1) You can have a look at these links for some software to help you, though I've not used most of them myself. But given that you've got timestamps on the files that were altered, start by looking at the log files for those dates and time. Grep for the filename of one of the files altered; that should give you the ip address of the malefactor, and then you can start grepping for that.

2 and 3) The first thing you need to do is to find out what is normal for your own site. One way to do that is to have a standard analyzing tool, such as awstat, go through your logs daily. Your main problem now appears to be that you don't know what is normal for your site, so you have no way of knowing what is abnormal.

4) There are ways of preventing some attacks, though of course it's an arms race. But just by removing the most common exploits you'll make it far less likely that you'll be the target of a random attack. You should make sure that your installation has the latest security patches for your OS, for apache and for php. You should also have a look at mod_security, which allows you to both log suspicious activity and to e.g. require that a client send headers such as User-Agent, which many crack tools don't.

Also, I'd advise you to consider sending your logs, especially error logs and any mod_security logs, to another server if possible, so that even if somebody manages to get into your server they won't be able to edit the log files to hide their tracks.