General way to monitor ALL memory usages
I have Kubuntu 16.04 and am using ZFS.
As you can see on superuser, I just had a long journey in tracking down where my memory was vanishing to.
It turned out to be the ARC of ZFS that was eating it up without showing up anywhere in the tools/files other than its own special file /proc/spl/kstat/zfs/arcstats
Is there a generic way to see how much memory is used by the kernel and if possible, by which kernel modules?
Example:
free -h was showing me this:
total used free shared buff/cache available
Mem: 31G 19G 1.8G 406M 10G 5.2G
Swap: 8.0G 0B 8.0G
So 19GB in use. But TOP only showed me about 8GB of used memory. So I went in search of the other 11GB.
6GB turned out to be slab which can be seen in /proc/memstat and with nmon. But 5GB were still missing.
In the end I found that it had been consumed by the ARC of ZFS. But it was only listed in its own special info file. /proc/spl/kstat/zfs/arcstats
Neither /proc/memstat nor top, htop, nmon, free, or atop could tell me who is using those 5GB.
Is there a file/tool to see the memory used by such special modules, without knowing/understanding those modules? Or do I have to account for every existing module if I would want to write a monitoring tool that can tell me this?
From my perspective, memory info under linux is a total mess. Dozen of number that all don't add up to total memory. I bet free calculates the "in use" parameter by substracting free/cache/buffers from the total.
I would like to have/write a tool/info where I can see
Total memory: 32GB
in use: 19GB
processes: 8GB
slab: 5GB
mapper: 1Gb
kernel modules: 5Gb
kernel: 0.3GB
spl: 4.7GB
cache: 6GB
buffers: 2GB
free: 5GB
Additional stats:
Active memory: 8.3GB
Active(anon): 7.1GB
Active(file): 1.2GB
Inactive memory: 2.4GB
...
You know, where each subsection actually addes up to its parent section and each byte of memory is accounted for ;)
So is such a tool possible under Linux? Does it exist already?
Update 1
The tool smem as suggested below indeed shows a colum "Dynamic kernel memory" which sums up to 100% of memory and accounts for ARC. However according to this answer
BTW: I already checked smem sources, it basically does (memtotal - userspace - free - cache).
So this is only achieved by simply substracting all known figures from /proc/memstat and labeling what is missing as "noncache Dynamic Kernel Memory" I find this disturbing and really hope the linux kernel has more to offer than that.
You might try smem for at least a goodly portion of what you are looking for.
It's available in the Universe repository and can be installed with sudo apt-get install smem
There are a number of useful switches that are covered on the man page man smem
Some examples are as follows: Note I'm using the -t switch for totals in all examples for brevity. you can leave it out if you want more verbose output.
PSS is the interesting column in these examples as it takes shared memory into account.
Unlike RSS it's meaningful to add it up.
System Wide
me@zippy-64bit:~$ smem -tw
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 989656 860076 129580
userspace memory 936032 180020 756012
free memory 5692716 5692716 0
----------------------------------------------------------
7618404 6732812 885592
By user
me@zippy-64bit:~$ smem -tu
User Count Swap USS PSS RSS
me 67 0 861452 898915 1245592
---------------------------------------------------
67 0 861452 898915 1245592
Also of note are smemcap for capture of data for future analysis with smem and smemstat which reports the physical memory usage taking into consideration shared memory. The tool can either report a current snapshot of memory usage or periodically dump out any changes in memory.
There's also a list at the bottom of the page of similar packages.
Sources:
- Correctly determining memory usage in Linux - Unix & Linux
man smem- Ubuntu – Details of package smem in xenial
- Smemstat - shared memory statistics
Alternative 1: You might also consider parsing the output of cat /proc/meminfo or if you want it by the page less /proc/meminfo
Much of the information here is used by free, top and ps commands. In fact, the output of the free command is similar in appearance to the contents and structure of /proc/meminfo. But by looking directly at /proc/meminfo, many more details are revealed including :
$ cat /proc/meminfo | grep "Slab"
Slab: 267680 kB
Alternative Source: https://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-proc-topfiles.html
Alternative 2: If you are truly committed to a better parsing of kernel RAM you might roll your own kstat. While kstat appears to be a Solaris only command. the link above will take you to sources for for an open source attempt at kstat that was last modified in 2013 (as of this edit.) A clever C programmer might be able to modify the code for their specific purpose.
Alternative 2 source: