What is using my memory on my Amazon EC2 instance?

I am currently running a single GPU Cluster Compute node on Amazon EC2 which is listed as having 22 GB of RAM per instance. top confirms that the instance does indeed have 22 GB of RAM but on an idle systems it appears that 5-6 GB of RAM are consistently in use. Neither top nor ps aux can identify the processes to which this usage is attributable.

5-6 GB seems too large for the OS to me, but perhaps I am wrong. Could this be an overhead due to virtualization on Amazon? Any other ideas or suggestions as to how I could track this down?

Solution 1:

The reason why you see a substantial amount of memory being used is that Linux is using much of it for file caching.

This would probably clarify things for you:

http://www.linuxatemyram.com/

Solution 2:

When you look at top, the free memory you see is the value before kernel caching.

top - 12:59:25 up 1 day, 15:47,  5 users,  load average: 0.04, 0.10, 0.12
Tasks: 161 total,   1 running, 160 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.5%us,  1.8%sy,  0.0%ni, 94.2%id,  0.4%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   3985464k total,  3369908k used,   615556k free,   752964k buffers
Swap:  1951740k total,       12k used,  1951728k free,  1439160k cached

My laptop as 3985464 kilo-bytes of memory total. But the linux kernel would cache most of that internally, so top only shows 619748kB of free memory. But if you look at it using the free command, you understand that, in fact, most of the memory is available... in the cache.

$ free
             total       used       free     shared    buffers     cached
Mem:       3985464    3365716     619748          0     752796    1439152
-/+ buffers/cache:    1173768    2811696
Swap:      1951740         12    1951728

The real free memory is the one shown on the buffers/cache line.

             total       used       free     shared    buffers     cached
-/+ buffers/cache:    1173768    2811696

This machine as 2.8GB of free memory.