How to find what is using linux swap or what is in the swap?
I have virtual linux (Fedora 17) server with 28GB RAM and 2GB swap. The server is running a MySQL DB that is set up to use most of the RAM.
After some time running the server starts to use swap to swap out unsued pages. That is fine as my swappiness is at default 60 and it is the expected behaviour.
The strange thing is that the number in top/meminfo does not correspond with info from processes. I.e. the server is reporting these numbers:
/proc/meminfo:
SwapCached: 24588 kB
SwapTotal: 2097148 kB
SwapFree: 865912 kB
top:
Mem: 28189800k total, 27583776k used, 606024k free, 163452k buffers
Swap: 2097148k total, 1231512k used, 865636k free, 6554356k cached
If I use the script from https://serverfault.com/a/423603/98204 it reports reasonable numbers (few MBs swapped by bash'es, systemd, etc) and one big allocation from MySQL (I omitted a lot of output lines):
892 [2442] qmgr -l -t fifo -u
896 [2412] /usr/libexec/postfix/master
904 [28382] mysql -u root
976 [27559] -bash
984 [27637] -bash
992 [27931] SCREEN
1000 [27932] /bin/bash
1192 [27558] sshd: admin@pts/0
1196 [27556] sshd: admin [priv]
1244 [1] /usr/lib/systemd/systemd
9444 [26626] /usr/bin/perl /bin/innotop
413852 [31039] /usr/libexec/mysqld --basedir=/usr --datadir=/data/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/data/mysql/err --open-files-limit=8192 --pid-file=/data/mysql/pid --socket=/data/mysql/mysql.sock --port=3306
449264 Total Swap Used
So if I get the script output right the total swap usage should be 449264K = ca. 440MB with mysql using ca. 90% of the swap.
The question is why this differs so much from the top and meminfo numbers? Is there any way how to "dump" swap info to see what is actually in it instead of summing the swap usages from all processes?
When analyzing the issue I came up with different ideas but they all seems to be wrong:
- The script output is not in KB. Even if it would be in 512 or 4KB units it won't match. Actually the ratio (1200:440) is about 3:1 which is "strange" number.
- There are some pages in swap that are somehow shared between processes as mentioned in https://serverfault.com/a/477664/98204 . If this is true how can I find the actual number of memory used like this? I mean it would need to make cca 800MB difference. And that does not sound right in this scenario.
- There are some "old" pages in swap used by processes that already finished. I would not mind that if I were able to find out how much is this "freeable" swap.
- There are pages in swap that has been swapped back to memory and are in swap just in case they did not change in RAM and need to be swapped out again as mentioned in https://serverfault.com/a/100636/98204 . But the SwapCached value is only 24MB.
The strange thing is that the swap usage is slowly increasing while the sum output from the script is roughly the same. In last 3 days the swap used increased from 1100MB to current 1230MB while the sum increased from 430MB to current 449MB (ca.).
The server has enough free(able) RAM so I could just turn off the swap and turn it back on. Or I could probably set swappiness to 0 so the swap would get used only if the is no other way. But I would like to solve the issue or at least find out what is the cause of this.
Solution 1:
Fedora 18 and up have smem in the repos. You could download the python script and install from source.
Here's a sample output (somewhat snipped and anonymized) from my machine:
# smem -s swap -t -k -n
PID User Command Swap USS PSS RSS
20917 1001 bash 0 1.1M 1.1M 1.9M
28329 0 python /bin/smem -s swap -t 0 6.3M 6.5M 7.4M
2719 1001 gnome-pty-helper 16.0K 72.0K 73.0K 516.0K
619 0 @sbin/mdadm --monitor --sca 28.0K 72.0K 73.0K 248.0K
[big snip]
32079 42 gnome-shell --mode=gdm 41.9M 1.9M 2.0M 5.0M
32403 1001 /opt/google/chrome/chrome - 43.1M 118.5M 119.4M 132.3M
4844 1002 /opt/google/chrome/chrome 48.1M 38.1M 41.9M 51.9M
5411 1002 /opt/google/chrome/chrome - 54.6M 33.4M 33.5M 36.8M
5624 1002 /opt/google/chrome/chrome - 72.4M 54.9M 55.5M 65.7M
24328 1002 /opt/Adobe/Reader9/Reader/i 77.5M 1.9M 2.0M 5.2M
4921 1002 /opt/google/chrome/chrome - 147.2M 258.4M 259.4M 272.0M
-------------------------------------------------------------------------------
214 14 1.1G 1.1G 1.2G 1.7G
The source also provides smemcap that will store all the relevant data so that smem can be run on it later.
To capture memory statistics on resource-constrained systems, the the smem source includes a utility named smemcap. smemcap captures all /proc entries required by smem and outputs them as an uncompressed .tar file to STDOUT. smem can analyze the output using the --source option. smemcap is small and does not require Python.
Solution 2:
You should check this script on another machine, because my system show correct swap usage:
# Your_script.sh
111280 Total Swap Used
# free
Swap: 33551716 120368 33431348
Very nearby 111280 ~= 120368.
Also, look at this script:
for proc in /proc/*; do cat $proc/smaps 2>/dev/null | awk '/Swap/{swap+=$2}END{print swap "\t'
readlink $proc/exe'"}'; done | sort -n | awk '{total+=$1}/[0-9]/;END{print total "\tTotal"}'
From this thread:
https://unix.stackexchange.com/questions/71714/linux-total-swap-used-swap-used-by-processes