Mysterious memory usage in Linux

We have some 64 bits Linux boxes (RHEL6) running on Microsoft Windows 2012 Server (Hypervisor) with the same problem. Those Linux guest servers are running in MS Windows Cloud (Hyper-V) with 16 hosts servers with 256 GB of RAM each one.

They start with the following memory usage:

# free -m
             total       used       free     shared    buffers     cached
Mem:         48259        653      47606          0         19        106
-/+ buffers/cache:        527      47732
Swap:        13999          0      13999



# cat /proc/meminfo
MemTotal:       49418204 kB
MemFree:        48749868 kB
Buffers:           20080 kB
Cached:           108564 kB
SwapCached:            0 kB
Active:           149652 kB
Inactive:          98856 kB
Active(anon):     120124 kB
Inactive(anon):     1884 kB
Active(file):      29528 kB
Inactive(file):    96972 kB
Unevictable:           4 kB
Mlocked:               4 kB
SwapTotal:      14335992 kB
SwapFree:       14335992 kB
Dirty:               788 kB
Writeback:             0 kB
AnonPages:        122196 kB
Mapped:            39844 kB
Shmem:              2132 kB
Slab:              51832 kB
SReclaimable:      14696 kB
SUnreclaim:        37136 kB
KernelStack:        5656 kB
PageTables:        15840 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    39045092 kB
Committed_AS:     490856 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      131964 kB
VmallocChunk:   34359602252 kB
HardwareCorrupted:     0 kB
AnonHugePages:     28672 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8128 kB
DirectMap2M:    50323456 kB

(Ordered by RES)

# top
top - 11:26:52 up 1 min,  1 user,  load average: 1.71, 0.73, 0.27
Tasks: 609 total,   1 running, 608 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.1%hi,  0.0%si,  0.0%st
Mem:  49418204k total,   674472k used, 48743732k free,    20472k buffers
Swap: 14335992k total,        0k used, 14335992k free,   111720k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1859 root      20   0  103m  28m 5784 S  0.0  0.1   0:00.85 Xvnc
 2037 root      20   0  508m  25m 8876 S  0.7  0.1   0:01.18 gnome-settings-
 2064 root      20   0  373m  17m  12m S  0.0  0.0   0:01.11 nautilus
 2141 root      20   0  464m  16m  12m S  0.0  0.0   0:00.14 clock-applet
 2063 root      20   0  319m  14m  10m S  0.0  0.0   0:00.28 gnome-panel
 2082 root      20   0  307m  12m 9100 S  0.0  0.0   0:00.11 nm-applet
 2139 root      20   0  381m  12m 9748 S  0.0  0.0   0:00.08 gdm-user-switch
 2093 root      20   0  442m  11m 9104 S  0.0  0.0   0:00.13 gnome-volume-co
 2116 root      20   0  299m  11m 9476 S  0.0  0.0   0:00.10 wnck-applet
 2118 root      20   0  307m  11m 8768 S  0.0  0.0   0:00.06 trashapplet
...

And 6 minutes later the memory is consumed by an unknown process or by the kernel:

uptime ; free -m
 11:31:52 up 6 min,  1 user,  load average: 1.05, 0.93, 0.47
             total       used       free     shared    buffers     cached
Mem:         48259      25296      22963          0         21        160
-/+ buffers/cache:      25115      23144
Swap:        13999          0      13999

]# cat /proc/meminfo
MemTotal:       49418204 kB
MemFree:        23514240 kB
Buffers:           21600 kB
Cached:           164428 kB
SwapCached:            0 kB
Active:           210768 kB
Inactive:         108108 kB
Active(anon):     133036 kB
Inactive(anon):     2332 kB
Active(file):      77732 kB
Inactive(file):   105776 kB
Unevictable:           8 kB
Mlocked:               8 kB
SwapTotal:      14335992 kB
SwapFree:       14335992 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        132892 kB
Mapped:            41976 kB
Shmem:              2516 kB
Slab:              52624 kB
SReclaimable:      17628 kB
SUnreclaim:        34996 kB
KernelStack:        5752 kB
PageTables:        15756 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    39045092 kB
Committed_AS:     654848 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      131964 kB
VmallocChunk:   34359602252 kB
HardwareCorrupted:     0 kB
AnonHugePages:     45056 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8128 kB
DirectMap2M:    50323456 kB

(Ordered by RES)

# top
top - 11:32:45 up 7 min,  1 user,  load average: 1.02, 0.94, 0.50
Tasks: 607 total,   1 running, 606 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.1%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  49418204k total, 25904096k used, 23514108k free,    21600k buffers
Swap: 14335992k total,        0k used, 14335992k free,   164428k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1859 root      20   0  103m  28m 6176 S  0.0  0.1   0:00.87 Xvnc
 2037 root      20   0  508m  25m 8916 S  0.0  0.1   0:01.19 gnome-settings-
 2064 root      20   0  373m  17m  12m S  0.0  0.0   0:01.61 nautilus
 2141 root      20   0  464m  16m  12m S  0.0  0.0   0:00.15 clock-applet
 2063 root      20   0  319m  14m  10m S  0.0  0.0   0:00.29 gnome-panel
 1788 root      20   0  779m  13m 5808 S  0.0  0.0   0:00.18 scxcimserver
 2082 root      20   0  307m  12m 9100 S  0.0  0.0   0:00.11 nm-applet
 2139 root      20   0  381m  12m 9748 S  0.0  0.0   0:00.08 gdm-user-switch
 2093 root      20   0  442m  11m 9104 S  0.0  0.0   0:00.13 gnome-volume-co
 2116 root      20   0  299m  11m 9476 S  0.0  0.0   0:00.10 wnck-applet
...

The sum of RSS memory reported by ps in MB is about 388:

# ps aux | awk '{sum+=$6} END {print sum / 1024}'
387.898

What else can I check to find out what is consuming the memory?


It sounds like Hyper-V is stealing the memory because your host is over-provisioned, a process called memory ballooning (or Dynamic Memory in the Hyper-V world).