systemd Using 4GB RAM After 18 Days of Uptime
I have a web server running CentOS 7, on which the systemd process is using nearly 4 GB of RAM after a couple weeks of uptime. RAM usage is increasing steadily at about 200MB per day. This and related processes like systemd-logind and dbus-daemon also use a sizable chunk of CPU much of the time. My other CentOS 6 server using "init" instead of systemd has no such resource usage.
In the top example below, during normal web serving without other processes running, systemd, systemd-logind, systemd-journal, and dbus-daemon use a combined total 10.7% of a quad-core CPU, and systemd is consuming 19% of the system's 16GB of RAM. This is not normal behavior, and after searching around I haven't found anyone else with this issue. What could cause this resource hogging? Any suggestions would be appreciated.
Output from top during an idle period (except for web serving):
top - 08:51:31 up 16 days, 13:43, 2 users, load average: 1.84, 1.39, 1.07
Tasks: 297 total, 2 running, 295 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.6 us, 3.6 sy, 0.0 ni, 90.6 id, 0.1 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 16212992 total, 2466564 free, 4275764 used, 9470664 buff/cache
KiB Swap: 4194300 total, 4070740 free, 123560 used. 10707392 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
743 dbus 20 0 27104 1856 1152 S 3.3 0.0 304:27.19 dbus-daemon
1 root 20 0 3247784 2.920g 1800 S 3.0 18.9 287:41.35 systemd
737 root 20 0 27416 2524 1304 S 2.7 0.0 225:32.66 systemd-logind
736 root 20 0 434760 3756 3076 S 2.0 0.0 172:26.53 NetworkManager
548 root 20 0 82276 34652 34516 S 1.7 0.2 160:20.16 systemd-journal
770 polkitd 20 0 522920 2956 2248 S 1.7 0.0 120:06.11 polkitd
716 root 16 -4 116744 1368 1312 S 1.3 0.0 93:26.54 auditd
3778 nginx 20 0 446488 14688 6564 S 1.3 0.1 2:18.80 php-fpm
3847 nginx 20 0 446316 14588 6548 S 1.3 0.1 2:19.29 php-fpm
7000 nginx 20 0 446132 14400 6544 S 1.3 0.1 1:22.77 php-fpm
14862 nginx 20 0 446304 14600 6580 S 1.3 0.1 1:32.25 php-fpm
30333 nginx 20 0 446292 14468 6528 S 1.3 0.1 1:40.78 php-fpm
740 root 20 0 784980 20112 19696 S 1.0 0.1 76:12.69 rsyslogd
3521 nginx 20 0 446188 14848 6748 S 1.0 0.1 2:20.00 php-fpm
3687 nginx 20 0 446036 14688 6764 S 1.0 0.1 2:20.45 php-fpm
3689 nginx 20 0 446408 14604 6552 S 1.0 0.1 2:19.75 php-fpm
3774 nginx 20 0 446288 14568 6552 S 1.0 0.1 2:19.68 php-fpm
3836 nginx 20 0 447416 15572 6564 S 1.0 0.1 2:21.06 php-fpm
4861 nginx 20 0 446260 14576 6540 S 1.0 0.1 2:18.94 php-fpm
4862 nginx 20 0 446508 15084 6764 S 1.0 0.1 2:20.71 php-fpm
13538 nginx 20 0 447204 15452 6572 S 1.0 0.1 1:32.33 php-fpm
15530 nginx 20 0 446292 14520 6528 S 1.0 0.1 1:32.55 php-fpm
28468 nginx 20 0 446356 14672 6568 S 1.0 0.1 1:42.21 php-fpm
29564 nginx 20 0 446292 14536 6548 S 1.0 0.1 1:41.11 php-fpm
30851 nginx 20 0 445956 14568 6748 S 1.0 0.1 1:49.66 php-fpm
Edit 2-14-16
I may have found something relevant in the output of "sudo journalctl" (see below). There are many lines occurring every second for hours at a time regarding SSH connections from one of my other production servers. These are rsync processes transferring files from the remote server to the server in question. This turns out to explain the CPU usage of systemd, systemd-logind, NetworkManager, and systemd-journal.
However, this could not explain the memory leak, which is the biggest problem. Since the original writing of this post a couple days ago, systemd has increased from 18.9% to 21.4% usage of system memory.
The log below has been modified to replace the real domain name and IP address of the servers.
Feb 14 10:02:13 hostname.domain.com systemd-logind[737]: New session 6467482 of user tropicg9.
Feb 14 10:02:13 hostname.domain.com systemd[1]: Started Session 6467482 of user tropicg9.
Feb 14 10:02:13 hostname.domain.com systemd[1]: Starting Session 6467482 of user tropicg9.
Feb 14 10:02:13 hostname.domain.com sshd[9665]: pam_unix(sshd:session): session opened for user tropicg9 by (uid=0)
Feb 14 10:02:13 hostname.domain.com sshd[9667]: Received disconnect from 1.2.3.4: 11: disconnected by user
Feb 14 10:02:13 hostname.domain.com sshd[9665]: pam_unix(sshd:session): session closed for user tropicg9
Feb 14 10:02:13 hostname.domain.com systemd-logind[737]: Removed session 6467482.
Feb 14 10:02:14 hostname.domain.com sshd[9728]: Accepted publickey for tropicg9 from 1.2.3.4 port 45289 ssh2: RSA 0b:
Feb 14 10:02:14 hostname.domain.com systemd-logind[737]: New session 6467483 of user tropicg9.
Feb 14 10:02:14 hostname.domain.com systemd[1]: Started Session 6467483 of user tropicg9.
Feb 14 10:02:14 hostname.domain.com systemd[1]: Starting Session 6467483 of user tropicg9.
Feb 14 10:02:14 hostname.domain.com sshd[9728]: pam_unix(sshd:session): session opened for user tropicg9 by (uid=0)
Feb 14 10:02:14 hostname.domain.com sshd[9735]: Received disconnect from 1.2.3.4: 11: disconnected by user
Feb 14 10:02:14 hostname.domain.com sshd[9728]: pam_unix(sshd:session): session closed for user tropicg9
Feb 14 10:02:14 hostname.domain.com systemd-logind[737]: Removed session 6467483.
Feb 14 10:02:15 hostname.domain.com sshd[9876]: Accepted publickey for tropicg9 from 1.2.3.4 port 45290 ssh2: RSA 0b:
Feb 14 10:02:15 hostname.domain.com systemd-logind[737]: New session 6467484 of user tropicg9.
Feb 14 10:02:15 hostname.domain.com systemd[1]: Started Session 6467484 of user tropicg9.
Feb 14 10:02:15 hostname.domain.com systemd[1]: Starting Session 6467484 of user tropicg9.
Feb 14 10:02:15 hostname.domain.com sshd[9876]: pam_unix(sshd:session): session opened for user tropicg9 by (uid=0)
Feb 14 10:02:15 hostname.domain.com sshd[9883]: Received disconnect from 1.2.3.4: 11: disconnected by user
Feb 14 10:02:15 hostname.domain.com sshd[9876]: pam_unix(sshd:session): session closed for user tropicg9
Feb 14 10:02:15 hostname.domain.com systemd-logind[737]: Removed session 6467484.
Feb 14 10:02:20 hostname.domain.com sshd[10333]: Accepted publickey for tropicg9 from 1.2.3.4 port 45291 ssh2: RSA 0b
Feb 14 10:02:20 hostname.domain.com systemd-logind[737]: New session 6467485 of user tropicg9.
Feb 14 10:02:20 hostname.domain.com systemd[1]: Started Session 6467485 of user tropicg9.
Feb 14 10:02:20 hostname.domain.com systemd[1]: Starting Session 6467485 of user tropicg9.
Feb 14 10:02:20 hostname.domain.com sshd[10333]: pam_unix(sshd:session): session opened for user tropicg9 by (uid=0)
Feb 14 10:02:20 hostname.domain.com sshd[10342]: Received disconnect from 1.2.3.4: 11: disconnected by user
Feb 14 10:02:20 hostname.domain.com sshd[10333]: pam_unix(sshd:session): session closed for user tropicg9
Feb 14 10:02:20 hostname.domain.com systemd-logind[737]: Removed session 6467485.
Feb 14 10:02:21 hostname.domain.com sshd[10450]: Accepted publickey for tropicg9 from 1.2.3.4 port 45292 ssh2: RSA 0b
Feb 14 10:02:21 hostname.domain.com systemd-logind[737]: New session 6467486 of user tropicg9.
Feb 14 10:02:21 hostname.domain.com systemd[1]: Started Session 6467486 of user tropicg9.
Feb 14 10:02:21 hostname.domain.com systemd[1]: Starting Session 6467486 of user tropicg9.
Feb 14 10:02:21 hostname.domain.com sshd[10450]: pam_unix(sshd:session): session opened for user tropicg9 by (uid=0)
Feb 14 10:02:21 hostname.domain.com sshd[10457]: Received disconnect from 1.2.3.4: 11: disconnected by user
Feb 14 10:02:21 hostname.domain.com sshd[10450]: pam_unix(sshd:session): session closed for user tropicg9
Feb 14 10:02:21 hostname.domain.com systemd-logind[737]: Removed session 6467486.
Feb 14 10:02:22 hostname.domain.com sshd[10473]: Accepted publickey for tropicg9 from 1.2.3.4 port 45293 ssh2: RSA 0b
Feb 14 10:02:22 hostname.domain.com systemd-logind[737]: New session 6467487 of user tropicg9.
Feb 14 10:02:22 hostname.domain.com systemd[1]: Started Session 6467487 of user tropicg9.
Feb 14 10:02:22 hostname.domain.com systemd[1]: Starting Session 6467487 of user tropicg9.
Feb 14 10:02:22 hostname.domain.com sshd[10473]: pam_unix(sshd:session): session opened for user tropicg9 by (uid=0)
Feb 14 10:02:22 hostname.domain.com sshd[10475]: Received disconnect from 1.2.3.4: 11: disconnected by user
Feb 14 10:02:22 hostname.domain.com sshd[10473]: pam_unix(sshd:session): session closed for user tropicg9
Feb 14 10:02:22 hostname.domain.com systemd-logind[737]: Removed session 6467487.
Feb 14 10:02:23 hostname.domain.com sshd[10484]: Accepted publickey for tropicg9 from 1.2.3.4 port 45294 ssh2: RSA 0b
Feb 14 10:02:23 hostname.domain.com systemd-logind[737]: New session 6467488 of user tropicg9.
Feb 14 10:02:23 hostname.domain.com systemd[1]: Started Session 6467488 of user tropicg9.
Feb 14 10:02:23 hostname.domain.com systemd[1]: Starting Session 6467488 of user tropicg9.
Feb 14 10:02:23 hostname.domain.com sshd[10484]: pam_unix(sshd:session): session opened for user tropicg9 by (uid=0)
Feb 14 10:02:23 hostname.domain.com sshd[10486]: Received disconnect from 1.2.3.4: 11: disconnected by user
Feb 14 10:02:23 hostname.domain.com sshd[10484]: pam_unix(sshd:session): session closed for user tropicg9
Feb 14 10:02:23 hostname.domain.com systemd-logind[737]: Removed session 6467488.
Feb 14 10:02:39 hostname.domain.com sshd[10654]: Accepted publickey for tropicg9 from 1.2.3.4 port 45295 ssh2: RSA 0b
Feb 14 10:02:39 hostname.domain.com systemd[1]: Started Session 6467489 of user tropicg9.
Feb 14 10:02:39 hostname.domain.com systemd-logind[737]: New session 6467489 of user tropicg9.
Feb 14 10:02:39 hostname.domain.com systemd[1]: Starting Session 6467489 of user tropicg9.
Feb 14 10:02:39 hostname.domain.com sshd[10654]: pam_unix(sshd:session): session opened for user tropicg9 by (uid=0)
Feb 14 10:02:39 hostname.domain.com sshd[10656]: Received disconnect from 1.2.3.4: 11: disconnected by user
Feb 14 10:02:39 hostname.domain.com sshd[10654]: pam_unix(sshd:session): session closed for user tropicg9
Feb 14 10:02:39 hostname.domain.com systemd-logind[737]: Removed session 6467489.session 6467489.
Update 2-16-16
Here is output from systemd-cgtop showing resource usage for active control groups (scroll to the right). This shows all of the heavy resource usage under the "root" path. This does not seem to narrow it down, but perhaps this information could be helpful.
There are only 86 scope files and associated directories under /run/systemd/system/, up to 6 days old. There was an issue where these files were orphaned during SSH connections resulting in thousands of entries and high CPU load, but that is not happening here.
Path Tasks %CPU Memory Input/s Output/s
/ 296 30.5 11.3G 657.8K 893.0K
/system.slice/NetworkManager.service 1 - - - -
/system.slice/auditd.service 1 - - - -
/system.slice/crond.service 1 - - - -
/system.slice/dbus.service 1 - - - -
/system.slice/irqbalance.service 1 - - - -
/system.slice/lvm2-lvmetad.service 1 - - - -
/system.slice/mariadb.service 2 - - - -
/system.slice/nginx.service 10 - - - -
/system.slice/php-fpm.service 101 - - - -
/system.slice/polkit.service 1 - - - -
/system.slice/postfix.service 3 - - - -
/system.slice/rsyslog.service 1 - - - -
/system.slice/smartd.service 1 - - - -
/system.slice/sshd.service 2 - - - -
/system.slice/system-getty.slice/[email protected] 1 - - - -
/system.slice/systemd-journald.service 1 - - - -
/system.slice/systemd-logind.service 1 - - - -
/system.slice/systemd-udevd.service 1 - - - -
/system.slice/tuned.service 1 - - - -
/system.slice/wpa_supplicant.service 1 - - - -
/user.slice/user-1000.slice/session-7170741.scope 4 - - - -
Temporary Clearing of systemd Memory
It appears that running systemctl daemon-reexec will release all memory allocated to the PID 1 process. However, the leak continues. A stop-gap solution to this problem is to set a daily cron to clear the memory, but it does not fix the leak. I've submitted a bug to Redhat since this is the stable release version of systemd for CentOS 7.x. Hopefully the leak can be found and plugged.
Check the trace of systemd process for mmap/mmunmap calls. It should reveal the problem:
yum install strace strace -ff -p 1
It's a quick and dirty way to diagnose memory leaks. Strace of systemd process should look similiar:
recvmsg(23, {msg_name(0)=NULL, msg_iov(1)=[{"WATCHDOG=1", 4096}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=620, uid=0, gid=0}}, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 10
open("/proc/620/cgroup", O_RDONLY|O_CLOEXEC) = 20
fstat(20, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcfd734e000
read(20, "10:cpuset:/\n9:perf_event:/\n8:hug"..., 1024) = 164
close(20) = 0
munmap(0x7fcfd734e000, 4096) = 0
It allocates memory, does something, than releases memory.
Checking the trace of system calls systemd does, you should discover where it can't finish calls and release allocated memory.
I suppose there's a problem with improperly mounted pseudo-filesystems or selinux, so systemd can't finish its calls.