Mysterious memory usage in Linux
We have some 64 bits Linux boxes (RHEL6) running on Microsoft Windows 2012 Server (Hypervisor) with the same problem. Those Linux guest servers are running in MS Windows Cloud (Hyper-V) with 16 hosts servers with 256 GB of RAM each one.
They start with the following memory usage:
# free -m
total used free shared buffers cached
Mem: 48259 653 47606 0 19 106
-/+ buffers/cache: 527 47732
Swap: 13999 0 13999
# cat /proc/meminfo
MemTotal: 49418204 kB
MemFree: 48749868 kB
Buffers: 20080 kB
Cached: 108564 kB
SwapCached: 0 kB
Active: 149652 kB
Inactive: 98856 kB
Active(anon): 120124 kB
Inactive(anon): 1884 kB
Active(file): 29528 kB
Inactive(file): 96972 kB
Unevictable: 4 kB
Mlocked: 4 kB
SwapTotal: 14335992 kB
SwapFree: 14335992 kB
Dirty: 788 kB
Writeback: 0 kB
AnonPages: 122196 kB
Mapped: 39844 kB
Shmem: 2132 kB
Slab: 51832 kB
SReclaimable: 14696 kB
SUnreclaim: 37136 kB
KernelStack: 5656 kB
PageTables: 15840 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 39045092 kB
Committed_AS: 490856 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 131964 kB
VmallocChunk: 34359602252 kB
HardwareCorrupted: 0 kB
AnonHugePages: 28672 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 8128 kB
DirectMap2M: 50323456 kB
(Ordered by RES)
# top
top - 11:26:52 up 1 min, 1 user, load average: 1.71, 0.73, 0.27
Tasks: 609 total, 1 running, 608 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.1%hi, 0.0%si, 0.0%st
Mem: 49418204k total, 674472k used, 48743732k free, 20472k buffers
Swap: 14335992k total, 0k used, 14335992k free, 111720k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1859 root 20 0 103m 28m 5784 S 0.0 0.1 0:00.85 Xvnc
2037 root 20 0 508m 25m 8876 S 0.7 0.1 0:01.18 gnome-settings-
2064 root 20 0 373m 17m 12m S 0.0 0.0 0:01.11 nautilus
2141 root 20 0 464m 16m 12m S 0.0 0.0 0:00.14 clock-applet
2063 root 20 0 319m 14m 10m S 0.0 0.0 0:00.28 gnome-panel
2082 root 20 0 307m 12m 9100 S 0.0 0.0 0:00.11 nm-applet
2139 root 20 0 381m 12m 9748 S 0.0 0.0 0:00.08 gdm-user-switch
2093 root 20 0 442m 11m 9104 S 0.0 0.0 0:00.13 gnome-volume-co
2116 root 20 0 299m 11m 9476 S 0.0 0.0 0:00.10 wnck-applet
2118 root 20 0 307m 11m 8768 S 0.0 0.0 0:00.06 trashapplet
...
And 6 minutes later the memory is consumed by an unknown process or by the kernel:
uptime ; free -m
11:31:52 up 6 min, 1 user, load average: 1.05, 0.93, 0.47
total used free shared buffers cached
Mem: 48259 25296 22963 0 21 160
-/+ buffers/cache: 25115 23144
Swap: 13999 0 13999
]# cat /proc/meminfo
MemTotal: 49418204 kB
MemFree: 23514240 kB
Buffers: 21600 kB
Cached: 164428 kB
SwapCached: 0 kB
Active: 210768 kB
Inactive: 108108 kB
Active(anon): 133036 kB
Inactive(anon): 2332 kB
Active(file): 77732 kB
Inactive(file): 105776 kB
Unevictable: 8 kB
Mlocked: 8 kB
SwapTotal: 14335992 kB
SwapFree: 14335992 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 132892 kB
Mapped: 41976 kB
Shmem: 2516 kB
Slab: 52624 kB
SReclaimable: 17628 kB
SUnreclaim: 34996 kB
KernelStack: 5752 kB
PageTables: 15756 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 39045092 kB
Committed_AS: 654848 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 131964 kB
VmallocChunk: 34359602252 kB
HardwareCorrupted: 0 kB
AnonHugePages: 45056 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 8128 kB
DirectMap2M: 50323456 kB
(Ordered by RES)
# top
top - 11:32:45 up 7 min, 1 user, load average: 1.02, 0.94, 0.50
Tasks: 607 total, 1 running, 606 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 49418204k total, 25904096k used, 23514108k free, 21600k buffers
Swap: 14335992k total, 0k used, 14335992k free, 164428k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1859 root 20 0 103m 28m 6176 S 0.0 0.1 0:00.87 Xvnc
2037 root 20 0 508m 25m 8916 S 0.0 0.1 0:01.19 gnome-settings-
2064 root 20 0 373m 17m 12m S 0.0 0.0 0:01.61 nautilus
2141 root 20 0 464m 16m 12m S 0.0 0.0 0:00.15 clock-applet
2063 root 20 0 319m 14m 10m S 0.0 0.0 0:00.29 gnome-panel
1788 root 20 0 779m 13m 5808 S 0.0 0.0 0:00.18 scxcimserver
2082 root 20 0 307m 12m 9100 S 0.0 0.0 0:00.11 nm-applet
2139 root 20 0 381m 12m 9748 S 0.0 0.0 0:00.08 gdm-user-switch
2093 root 20 0 442m 11m 9104 S 0.0 0.0 0:00.13 gnome-volume-co
2116 root 20 0 299m 11m 9476 S 0.0 0.0 0:00.10 wnck-applet
...
The sum of RSS memory reported by ps in MB is about 388:
# ps aux | awk '{sum+=$6} END {print sum / 1024}'
387.898
What else can I check to find out what is consuming the memory?
It sounds like Hyper-V is stealing the memory because your host is over-provisioned, a process called memory ballooning (or Dynamic Memory in the Hyper-V world).