Ubuntu 14.04 /tmp/systemd process using 100% CPU
We have a live AWS EC2 instance with Ubuntu 14.04, which I have been handed to maintain. An oddity that I have found is that I have a process of /tmp/systemd using all available CPU:
top - 11:35:20 up 2:34, 1 user, load average: 1.13, 1.16, 1.15
Tasks: 114 total, 2 running, 111 sleeping, 0 stopped, 1 zombie
%Cpu(s): 32.7 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 67.3 st
KiB Mem: 2048516 total, 726492 used, 1322024 free, 38784 buffers
KiB Swap: 0 total, 0 used, 0 free. 439788 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1797 www-data 20 0 192900 4008 1140 S 95.2 0.2 107:42.82 /tmp/systemd
1145 mysql 20 0 681596 121328 7560 S 3.6 5.9 2:35.84 /usr/sbin/mysqld
3669 ubuntu 20 0 103084 1904 920 S 0.7 0.1 0:00.06 sshd: ubuntu@pts/0
40 root rt 0 0 0 0 S 0.3 0.0 0:04.96 [watchdog/0]
3692 ubuntu 20 0 23728 1652 1104 R 0.3 0.1 0:00.11 top
1 root 20 0 33556 2880 1480 S 0.0 0.1 0:03.20 /sbin/init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [kthreadd]
I thought the /tmp/systemd seemed a tad odd for a command source, so I checked the /tmp directory - and it's completely empty.
Running an strace seems to indicate that it is polling for something to finish:
ubuntu@ip-10-0-0-157:~$ sudo strace -fvvp 1797
Process 1797 attached with 6 threads
[pid 1801] futex(0x799404, FUTEX_WAIT_PRIVATE, 4, NULL <unfinished ...>
[pid 1802] futex(0x799404, FUTEX_WAIT_PRIVATE, 4, NULL <unfinished ...>
[pid 1803] futex(0x799404, FUTEX_WAIT_PRIVATE, 4, NULL <unfinished ...>
[pid 1800] futex(0x799404, FUTEX_WAIT_PRIVATE, 4, NULL <unfinished ...>
[pid 1797] clock_gettime(CLOCK_MONOTONIC, {9448, 290556741}) = 0
[pid 1797] epoll_wait(7, <unfinished ...>
[pid 1799] sched_yield() = 0
[pid 1799] clock_gettime(CLOCK_REALTIME, {1524656311, 141773213}) = 0
[pid 1797] <... epoll_wait resumed> {}, 1024, 27) = 0
[pid 1797] clock_gettime(CLOCK_MONOTONIC, {9448, 328709201}) = 0
[pid 1797] clock_gettime(CLOCK_MONOTONIC, {9448, 328773526}) = 0
[pid 1797] epoll_wait(7, <unfinished ...>
[pid 1799] sched_yield() = 0
[pid 1799] sched_yield() = 0
[pid 1797] <... epoll_wait resumed> {}, 1024, 500) = 0
[pid 1797] clock_gettime(CLOCK_MONOTONIC, {9448, 839857928}) = 0
[pid 1797] clock_gettime(CLOCK_MONOTONIC, {9448, 839934892}) = 0
[pid 1797] epoll_wait(7, {}, 1024, 18) = 0
[pid 1797] clock_gettime(CLOCK_MONOTONIC, {9448, 860012749}) = 0
[pid 1797] clock_gettime(CLOCK_MONOTONIC, {9448, 860081346}) = 0
[pid 1797] epoll_wait(7, <unfinished ...>
[pid 1799] sched_yield() = 0
[pid 1799] sched_yield() = 0
[pid 1799] sched_yield() = 0
[pid 1799] sched_yield() = 0
[pid 1797] <... epoll_wait resumed> {}, 1024, 479) = 0
[pid 1797] clock_gettime(CLOCK_MONOTONIC, {9449, 350734613}) = 0
[pid 1797] clock_gettime(CLOCK_MONOTONIC, {9449, 350806967}) = 0
[pid 1797] epoll_wait(7, <unfinished ...>
[pid 1799] sched_yield() = 0
[pid 1799] sched_yield() = 0
[pid 1799] sched_yield() = 0
q[pid 1797] <... epoll_wait resumed> {}, 1024, 500) = 0
[pid 1797] clock_gettime(CLOCK_MONOTONIC, {9449, 905946457}) = 0
[pid 1797] clock_gettime(CLOCK_MONOTONIC, {9449, 906007520}) = 0
[pid 1797] epoll_wait(7, <unfinished ...>
[pid 1799] sched_yield() = 0
[pid 1799] sched_yield() = 0
I've tried updating and rebooting the server, but it changes nothing. I can kill the process, but approximately 1 hour later it will restart.
Can you please advise on how to locate source of this or how I can find out what it is doing and what is starting it?
Solution 1:
There is no /tmp/systemd binary or temporary binary, given that the user which started the process is non-interactive "www-data" I would suggest that your webserver was compromised.
Check for any suspicions outbound connections, re-mount /tmp with noexec and check if everything owned by www-data is as it should be. Look for shell scripts or other executables that should not belong to www-data.
Looks like Tiny XMR mooner - https://xorl.wordpress.com/2017/12/21/the-tiny-xml-mooner-linux-cryptominer-malware/
Best practice is such cases is to make image of the machine (with dd for example) and analyze it offline in environment with no Internet access.