`ps` and `bin/bash su` in ps aux output
I have a shared server (amazon ec2 micro powered by bitnami) with a web application (Wordpress), which is showing low performance lately (with relatively stable traffic and usage). The only suspitious lines in apache2/logs/access.log I see are of form
$ tail --lines=1000 /opt/bitnami/apache2/logs/access_log
93.120.84.31 - - [06/Nov/2013:03:02:54 +0000] "POST /cgi-bin/php-cgi?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C%61%74%69%6F%6E%3D%6F%6E+%2D%64+%64%69%73%61%62%6C%65%5F%66%75%6E%63%74%69%6F%6E%73%3D%22%22+%2D%64+%6F%70%65%6E%5F%62%61%73%65%64%69%72%3D%6E%6F%6E%65+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%64+%63%67%69%2E%66%6F%72%63%65%5F%72%65%64%69%72%65%63%74%3D%30+%2D%64+%63%67%69%2E%72%65%64%69%72%65%63%74%5F%73%74%61%74%75%73%5F%65%6E%76%3D%30+%2D%6E HTTP/1.1" 504 249
and in ps aux
output
daemon 29369 0.0 0.2 17688 1500 ? S 10:52 0:00 /bin/bash ./su 57.54
daemon 29377 30.5 0.0 1780 544 ? R 10:52 3:14 ./ps 57.54 22
what are those processes, and if this is a [known] exploit/vulnerability, how do I close it? I googled this mention, but can't find out whether this is applicable in my case.
If this question belongs to other SE site, please help me ask it where appropriate.
Many thanks in advance!
UPDATE
Followed tips from @Ladadadada, results are below. Yep, it's intrusion. Server stopped, restarted from weekly backup (all db and other things were on separate drives and I hope wasn't compromized) and being monitored. All suggestions on what's going on and how to prevent this in future are appreciated.
for ps
proc:
$ sudo ls -l /proc/30633/fd
total 0
lr-x------ 1 daemon daemon 64 2013-11-06 13:05 0 -> /dev/null
l-wx------ 1 daemon daemon 64 2013-11-06 13:05 1 -> pipe:[2010646]
l-wx------ 1 daemon daemon 64 2013-11-06 13:05 2 -> pipe:[1918347]
lr-x------ 1 daemon daemon 64 2013-11-06 13:05 255 -> /tmp/.ssh_auth/.b/.ssh_auth/.b/su
lr-x------ 1 daemon daemon 64 2013-11-06 13:05 3 -> /opt/bitnami/apache2/cgi-bin/php-cgi
for /bin/bash su
proc
$ sudo ls -l /proc/30641/fd | more
total 0
lr-x------ 1 daemon daemon 64 2013-11-06 13:04 0 -> /dev/null
l-wx------ 1 daemon daemon 64 2013-11-06 13:04 1 -> pipe:[2010646]
lrwx------ 1 daemon daemon 64 2013-11-06 13:04 10 -> socket:[5538939]
lrwx------ 1 daemon daemon 64 2013-11-06 13:04 100 -> socket:[5539029]
lrwx------ 1 daemon daemon 64 2013-11-06 13:04 101 -> socket:[5539030]
lr-x------ 1 daemon daemon 64 2013-11-06 13:04 3 -> /opt/bitnami/apache2/cgi-bin/php-cgi
and many more. Contents of temp folder:
/tmp/.ssh_auth/.b/.ssh_auth/.b$ ll
total 1848
drwxr-xr-x 2 daemon daemon 4096 2013-11-06 13:04 ./
drwxr-xr-x 3 daemon daemon 4096 2013-11-06 01:34 ../
-rw-r--r-- 1 daemon daemon 0 2013-11-06 13:04 246.120.pscan.22
-rw-r--r-- 1 daemon daemon 0 2013-11-06 03:42 .a
-rwxr-xr-x 1 daemon daemon 1384518 2005-06-05 20:24 brute*
-rwxr-xr-x 1 daemon daemon 1161 2013-03-13 20:22 dns-pool*
-rwxr-xr-x 1 daemon daemon 73 2013-11-05 04:14 pass.txt*
-rwxr-xr-x 1 daemon daemon 154 2013-11-05 04:13 print*
-rwxr-xr-x 1 daemon daemon 16071 2012-08-12 16:19 ps*
-rwxr-xr-x 1 daemon daemon 453972 2011-03-21 15:15 ss*
-rwxr-xr-x 1 daemon daemon 520 2013-11-06 01:33 su*
UPDATE 2
Winner by split second with Amazon, just after instance shutdown:
Hello,
We have detected that your instance(s):
i-xxxxxxxx
have been behaving in the following way that is against our AWS Customer Agreement:
Port Scanning
Solution 1:
That long string in your access logs is URL encoded and decodes to this:
-d allow_url_include=on -d safe_mode=off -d suhosin.simulation=on -d disable_functions="" -d open_basedir=none -d auto_prepend_file=php://input -d cgi.force_redirect=0 -d cgi.redirect_status_env=0
It looks to me like it's passing a bunch of options to php-cgi
that turn off all the security features such as disallowing including URLs, disallowing certain functions and safe mode. It also allows PHP code to be passed in on STDIN which probably allows the attacker to simply supply code to be run as the POST parameters.
That's almost certainly malicious.
The vulnerability in this case is probably that php-cgi
is in a web-accessible directory and/or can be requested directly.
The two lines in your ps
output are also suspicious. You might investigate them further by looking in the /proc
filesystem.
sudo ls -l /proc/29377/fd
Or by using strace
on the PID:
sudo strace -f -p 29377 -s400
The output from strace
can be quite large and difficult to read but it shouldn't be too hard to spot any spam or network connection attempts in there. You can limit the output with options like -e trace=file
or -e trace=file,network
.
Once you determine that it is malicious and you know how they got in, you should follow the instructions in this question to clean up afterwards. Simply removing the files you find is not enough because the attacker could have done anything to hide himself in the system.