How to do a post-mortem of a server hack
I have a Windows Server 2003 SP2 machine with IIS6, SQL Server 2005, MySQL 5 and PHP 4.3 installed on it. This is not a production machine, but it is exposed to the world via a domain name. Remote desktop is enabled on the machine and two administive accounts are active on it.
This morning I found that the machine had been logged off with an uknown user's name still in the login text box. Upon further investigation I've found that two windows users have been created, anti-virus has been uninstalled and a smattering of .exe files have been dropped into the C: drive.
What I would like to know is, what steps should I take to make sure that this does not happen again, and areas should I focus on to determine the avenue of entry. I've already checked netstat -a to see which ports are open, and nothing appears odd there. I did find unknown files in the data folder for MySQL which I'm thinking might have been the entry point but I'm not sure.
I'd really appreciate the steps to conduct a good post-mortem of a server hack so that I can avoid this in the future.
Post Investigation Review
After some investigation I think I found out what happened. First the machine has not been online during the time frame of August '08 to October '09. During that time frame a security vulnerability was discovered, the MS08-067 Vulnerability. "This is a remote code execution vulnerability. An attacker who successfully exploited this vulnerability could take complete control of an affected system remotely. On Microsoft Windows 2000-based, Windows XP-based, and Windows Server 2003-based systems, an attacker could exploit this vulnerability over RPC without authentication and could run arbitrary code." This vulerability was fixed with the KB958644 Security Update which came out in October 2008.
Because the machine was offline at the time and missed out on this update, I believe that this vulnerability was exploited soon after the machine came back online in October of '09. I found references to a bycnboy.exe program which has been described as a backdoor program which then creates much havoc on an infected system. Soon after the machine was online, automatic updates installed the patch which closed off the ability to have remote control of the system. Because the backdoor was now closed, I believe that the attacker then created physical accounts on the machine and was able to utilize the machine for a further week until I noticed what was happening.
After agressively pursing the malicious code, .exes and .dlls, removing self hosting websites and user accounts, the machine is now again in a working state. For the near future I will be monitoring the system and reviewing the server logs to determin if a repeat of the incident is happening.
Thank you for the information and steps that were provided.
Solution 1:
Doing a post-mortem is a black art in an of itself. It's a little different each time because really no two break-ins are the same. With that in mind, a basic overview of my recommended process is below, with a few specific notes toward your situation:
- Physically Disconnect the machine from the network. (Really. Do it now.)
- Optional Step: Make a binary image copy of the hard drive for future use.
- Make a copy of all log files, valuable data, etc. onto a removable hard drive
- Optionally copy any "hacker tools" you find as well
- Begin the actual post-mortem. In your case:
- Note any new or missing user accounts. See if their home folders have any "interesting" contents.
- Note any new or missing programs/binaries/data files.
- Check the MySQL logs first - Look for anything "unusual"
- Check the rest of the server logs. See if you can find the new users being created, addresses they logged in from, etc.
- Look for evidence of data damage or theft
- When you find the cause of the problem, note how to keep it from happening again.
- Wipe the server clean: Format and reinstall everything, restore your data & plug the original hole with your notes from #5.
You typically perform step 2 if you're going to be involving law enforcement. You perform Step 3 so you can review information after the server gets rebuilt without having to read the image copy you made in step 2.
How detailed step 4 gets depends on your goals: Just plugging the hole is a different kind of investigation than tracking down who stole some valuable bit of data :)
Step 6 is IMHO critical. You don't "fix" a compromised host: You wipe it and start over from a known good state. This ensures that you won't miss some nugget of nasty left on the box as a time bomb.
This is by no means a complete post-mortem outline. I'm marking this as community wiki as I'm always looking for improvements to the process - I don't use it often :-)