Memory usage doesn't add up over time
The memory usage on my Ubuntu 13.04 system doesn't add up over time. I've tried the 3.8.0-xx kernels and the latest 3.9 from /~kernel-ppa/mainline.
When I first boot, it all adds up correctly but after a couple of days, this is what it looks like.
$ free -k
total used free shared buffers cached
Mem: 7978976 7319920 659056 0 97248 1344356
-/+ buffers/cache: 5878316 2100660
Swap: 2438140 0 2438140
Free is reporting that about 5.8G of resident memory is being used excluding cache and buffers.
When I sum the rss reported by the ps command, the totals differ dramatically.
sudo ps -eo pid,rss,args > mem.txt
PID RSS COMMAND
6377 848,268 /usr/lib/firefox/firefox
6095 243,664 /usr/bin/gnome-shell
1752 72,300 /usr/sbin/mysqld
6167 71,460 /var/lib/dropbox/.dropbox-dist/dropbox
7608 66,952 /usr/bin/python /usr/bin/deluge-gtk
23320 57,892 /usr/bin/X :3 -background none -verbose -auth /run/gdm/auth-for-gdm-9zmczs/database -nolisten tcp vt7
6445 45,476 /usr/lib/firefox/plugin-container /usr/lib/adobe-flashplugin/libflashplayer.so -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 6377 true plugin
6159 39,784 nautilus -n
6153 37,276 /usr/bin/python /usr/bin/autokey-gtk
9306 29,432 gnome-terminal
6152 27,152 nm-applet
6066 24,796 /usr/lib/gnome-settings-daemon/gnome-settings-daemon
6434 20,120 /usr/lib/firefox/plugin-container /opt/google/talkplugin/libnpgoogletalk.so -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 6377 true plugin
9505 17,700 /usr/sbin/apache2 -k start
6154 16,096 update-notifier
6437 13,516 /opt/google/talkplugin/GoogleTalkPlugin
6184 13,192 /usr/lib/evolution/evolution-calendar-factory
5983 12,372 gnome-session --session=gnome
3746 9,468 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
9510 8,476 /usr/sbin/apache2 -k start
9511 8,476 /usr/sbin/apache2 -k start
9512 8,476 /usr/sbin/apache2 -k start
9513 8,476 /usr/sbin/apache2 -k start
9514 8,476 /usr/sbin/apache2 -k start
6112 7,732 /usr/lib/evolution/evolution-source-registry
1890 7,212 /usr/bin/atop -a -w /var/log/atop/atop_20130425 600
6106 6,912 /usr/lib/gnome-shell/gnome-shell-calendar-server
1646 6,868 NetworkManager
6073 6,404 /usr/bin/pulseaudio --start --log-target=syslog
6125 6,080 /usr/lib/telepathy/mission-control-5
5588 5,592 /usr/lib/udisks2/udisksd --no-debug
1229 5,568 /usr/lib/colord/colord
6129 5,560 /usr/lib/gvfs/gvfs-udisks2-volume-monitor
1205 5,532 smbd -F
5954 5,220 gdm-session-worker [pam/gdm-password]
1656 5,164 /usr/lib/policykit-1/polkitd --no-debug
9312 5,112 -bash
5965 5,040 /usr/bin/gnome-keyring-daemon --daemonize --login
9531 4,952 -bash
1759 4,900 whoopsie
1606 4,412 /usr/sbin/winbindd -F
3149 4,300 /usr/lib/upower/upowerd
2992 4,068 /usr/sbin/console-kit-daemon --no-daemon
23316 3,952 /usr/lib/gdm/gdm-simple-slave --display-id /org/gnome/DisplayManager/Displays/_3
6299 3,800 /usr/lib/gvfs/gvfsd-metadata
1090 3,748 /usr/sbin/cupsd -F
1721 3,708 /sbin/dhclient -d -sf /usr/lib/NetworkManager/nm-dhcp-client.action -pf /run/sendsigs.omit.d/network-manager.dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-681b428f-beaf-8932-dce4-687ed5bae28e-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0
2988 3,672 /usr/lib/accountsservice/accounts-daemon
6228 3,540 /usr/lib/gvfs/gvfsd-trash --spawner :1.4 /org/gtk/gvfs/exec_spaw/0
6043 3,296 /usr/lib/at-spi2-core/at-spi2-registryd --use-gnome-session
1728 3,288 gdm
6136 3,220 /usr/lib/gvfs/gvfs-gphoto2-volume-monitor
6144 3,212 /usr/lib/gvfs/gvfs-afc-volume-monitor
6036 3,204 /usr/lib/at-spi2-core/at-spi-bus-launcher
1607 3,196 /usr/sbin/modem-manager
1 3,096 /sbin/init
6087 3,044 /usr/lib/pulseaudio/pulse/gconf-helper
6047 2,992 /usr/lib/gvfs/gvfsd
6089 2,900 /usr/lib/x86_64-linux-gnu/gconf/gconfd-2
6274 2,832 /usr/lib/dconf/dconf-service
1248 2,772 /usr/sbin/cups-browsed
1571 2,696 /usr/sbin/sshd -D
6140 2,648 /usr/lib/gvfs/gvfs-mtp-volume-monitor
1484 2,608 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
9662 2,600 sudo ps -eo pid,rss,args
3614 2,428 /usr/sbin/afpd -U uams_dhx2.so,uams_clrtxt.so -g nobody -c 50 -n cheetah
771 2,340 dbus-daemon --system --fork
5376 2,124 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 117:126
6032 1,908 //bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
1642 1,780 /usr/sbin/winbindd -F
3460 1,760 qmgr -l -t fifo -u
928 1,728 avahi-daemon: running [cheetah.local]
866 1,704 /usr/sbin/bluetoothd
719 1,700 /sbin/udevd --daemon
9637 1,696 man ps
1627 1,684 nmbd -D
762 1,652 rsyslogd -c5
2850 1,636 /usr/lib/postfix/master
1239 1,608 smbd -F
7599 1,608 pickup -l -t fifo -u -c
6040 1,488 /bin/dbus-daemon --config-file=/etc/at-spi2/accessibility.conf --nofork --print-address 3
1760 1,472 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/NetworkManager/dnsmasq.pid --listen-address=127.0.1.1 --conf-file=/var/run/NetworkManager/dnsmasq.conf --cache-size=0 --proxy-dnssec --enable-dbus=org.freedesktop.NetworkManager.dnsmasq --conf-dir=/etc/NetworkManager/dnsmasq.d
5399 1,368 /sbin/udevd --daemon
5400 1,368 /sbin/udevd --daemon
1233 1,356 rpc.statd -L
3516 1,308 /usr/lib/rtkit/rtkit-daemon
9648 1,188 pager -s
1719 1,044 cron
1675 976 /sbin/getty -8 38400 tty5
1691 976 /sbin/getty -8 38400 tty2
1692 972 /sbin/getty -8 38400 tty3
1695 972 /sbin/getty -8 38400 tty6
3971 972 /sbin/getty -8 38400 tty1
1668 968 /sbin/getty -8 38400 tty4
1216 964 rpcbind -w
716 900 upstart-udev-bridge --daemon
2893 880 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -r /var/run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new
9663 868 ps -eo pid,rss,args
9311 824 gnome-pty-helper
1703 816 acpid -c /etc/acpi/events -s /var/run/acpid.socket
1737 776 /usr/sbin/irqbalance
3606 760 /usr/sbin/cnid_metad -l log_note
1235 732 upstart-socket-bridge --daemon
763 680 rpc.idmapd
6031 592 /usr/bin/dbus-launch --exit-with-session gnome-session --session=gnome
933 472 avahi-daemon: chroot helper
457 420 upstart-file-bridge --daemon
6028 316 /usr/bin/ssh-agent /usr/bin/dbus-launch --exit-with-session gnome-session --session=gnome
1717 168 atd
2 0 [kthreadd]
3 0 [ksoftirqd/0]
5 0 [kworker/0:0H]
7 0 [kworker/u:0H]
8 0 [migration/0]
9 0 [rcu_bh]
10 0 [rcu_sched]
11 0 [watchdog/0]
12 0 [watchdog/1]
13 0 [migration/1]
14 0 [ksoftirqd/1]
16 0 [kworker/1:0H]
17 0 [watchdog/2]
18 0 [migration/2]
19 0 [ksoftirqd/2]
21 0 [kworker/2:0H]
22 0 [watchdog/3]
23 0 [migration/3]
24 0 [ksoftirqd/3]
26 0 [kworker/3:0H]
27 0 [watchdog/4]
28 0 [migration/4]
29 0 [ksoftirqd/4]
31 0 [kworker/4:0H]
32 0 [watchdog/5]
33 0 [migration/5]
34 0 [ksoftirqd/5]
36 0 [kworker/5:0H]
37 0 [khelper]
38 0 [kdevtmpfs]
39 0 [netns]
40 0 [bdi-default]
41 0 [kintegrityd]
42 0 [kblockd]
43 0 [ata_sff]
44 0 [khubd]
45 0 [md]
46 0 [devfreq_wq]
48 0 [irq/72-AMD-Vi]
49 0 [khungtaskd]
50 0 [kswapd0]
51 0 [ksmd]
52 0 [khugepaged]
53 0 [fsnotify_mark]
54 0 [ecryptfs-kthrea]
55 0 [crypto]
66 0 [kthrotld]
74 0 [binder]
93 0 [deferwq]
94 0 [charger_manager]
198 0 [scsi_eh_0]
204 0 [scsi_eh_1]
205 0 [firewire]
219 0 [kworker/u:3]
220 0 [scsi_eh_2]
239 0 [scsi_eh_3]
251 0 [scsi_eh_4]
255 0 [scsi_eh_5]
259 0 [scsi_eh_6]
260 0 [scsi_eh_7]
269 0 [kworker/u:6]
283 0 [scsi_eh_8]
284 0 [scsi_eh_9]
360 0 [kworker/5:1H]
361 0 [jbd2/sda1-8]
362 0 [ext4-dio-unwrit]
643 0 [xfsalloc]
644 0 [xfs_mru_cache]
645 0 [xfslogd]
646 0 [xfs-data/sdb1]
647 0 [xfs-conv/sdb1]
648 0 [xfs-cil/sdb1]
650 0 [xfsaild/sdb1]
695 0 [xfs-data/sdc1]
696 0 [xfs-conv/sdc1]
697 0 [xfs-cil/sdc1]
698 0 [xfsaild/sdc1]
730 0 [rpciod]
743 0 [nfsiod]
922 0 [krfcommd]
944 0 [kworker/4:1H]
1017 0 [edac-poller]
1029 0 [hd-audio0]
1031 0 [kpsmoused]
1060 0 [kvm-irqfd-clean]
1107 0 [hd-audio1]
1157 0 [kworker/3:1H]
1190 0 [kworker/2:1H]
1198 0 [kworker/1:1H]
1793 0 [kauditd]
1845 0 [flush-8:0]
3569 0 [iprt]
4867 0 [kworker/4:1]
5403 0 [nfsv4.0-svc]
5848 0 [kworker/0:1H]
5935 0 [kworker/2:3]
9274 0 [kworker/5:2]
9527 0 [kworker/5:0]
9630 0 [flush-8:32]
10694 0 [kworker/3:3]
13358 0 [kworker/1:0]
18154 0 [kworker/0:0]
19625 0 [kworker/4:0]
23502 0 [kworker/3:2]
31403 0 [kworker/2:1]
32141 0 [kworker/0:2]
32655 0 [kworker/1:1]
1,939,900
ps reports that the sum off all the resident memory used by all my processes is about 1.9G.
What processes are using the remaining 3.9G?
EDIT: More information on kernel memory usage. slabtop reports the following.
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
2233988 2233988 100% 0.88K 62057 36 1985824K xfs_inode
735948 714094 97% 0.22K 20443 36 163544K xfs_ili
129280 117946 91% 1.00K 4040 32 129280K nfs_inode_cache
92344 92344 100% 0.93K 2716 34 86912K ext4_inode_cache
292866 292866 100% 0.19K 13946 21 55784K dentry
75096 71346 95% 0.55K 2682 28 42912K radix_tree_node
Does this mean that the kernel is using almost 2GB to cache xfs_inodes, cache that's not reported in the cached column of the free command?
Solution 1:
The memory has been found. It was in the xfs_inode cache which wasn't reported by ps or free. slabtop did however show me what I needed. The reason for such a large xfs_inode cache is because of a nightly backup. The memory can be released with the following command.
echo 3 > /proc/sys/vm/drop_caches
From http://www.linuxinsight.com/proc_sys_vm_drop_caches.html
To free pagecache:
echo 1 > /proc/sys/vm/drop_cachesTo free dentries and inodes:
echo 2 > /proc/sys/vm/drop_cachesTo free pagecache, dentries and inodes:
echo 3 > /proc/sys/vm/drop_caches
More information can be found on this blog.
http://serenadetoacuckooo.blogspot.ca/2012/03/linux-memory-tweaks-for-improved.html
I have to decide now on whether I want to drop the cache after backups, set vfs_cache_pressure back to 100 or just buy more memory.