kswapd0 is taking a lot of cpu

kswapd0 is taking 99.9% of my CPU as top shows me, the problem appeared today when gaming and first time it went away after 6 minutes and now it has been doing it for about 20 minutes. How is this fixable and what is causing this?

Solution 1:

The process kswapd0 is the process that manages virtual memory. Your machine should have RAM, SWAP, and the EXT4 on your HDD/SSD. The ext4 is where everything is stored, and it is always slower to access than RAM. RAM is like a half-way running space for programs to access information quickly. Most computers have at least 4GB of RAM, which under normal conditions is plenty. When playing a game, however, you may run low on RAM space, which is where SWAP comes in.

SWAP is a fake RAM located on your HDD/SSD next to your EXT4. It is quicker to access than the EXT4, but it is much slower than actual RAM. When you run low on memory, kswapd0 moves programs that you are not using/not using as much as other programs to the SWAP, which causes extreme lag on those processes. If your game was needing 5GB RAM, 1GB at LEAST would be in SWAP. That means when it tries to access that information, it has to wait longer to get it.

This entire process causes extreme CPU usage, moving information from and to SWAP and RAM and handling the request of information all at the same time. How to solve this issue?

Tell kswapd0 to only move stuff to SWAP when you are completely OUT of RAM. This is the single most effective method to resolving SWAP issues. Run

echo vm.swappiness=0 | sudo tee -a /etc/sysctl.conf

where 0 is the percent left out of 100 at which SWAP should be used (when you have 0% RAM left, SWAP will start taking in data). You can also just edit /etc/sysctl.conf to your liking instead of adding this command to the end of it everytime using gedit or nano or whatever, be sure to sudo though, this file is root owned. Reboot and your are set!
Reduce the consumption of RAM by other processes or close other programs while running high memory programs. This is why most games tell you to close all other windows before playing, or installations do the same. Things like file syncing services tend to take a lot of memory.
Buy more RAM. Installing RAM is not as hard as it sounds. One or two screws on a small compartment (if you are on a laptop) and a simple click. Just be sure you are buying the correct kind!
Lower processes of the CPU much as you did with the RAM. This will help those RAM to SWAP bursts to go much smoother.

That's the best that you can do. Others may say disable swap completely, but that is dangerous and I would NOT recommend that. That can cause entire systems to freeze up if there is a memory leak or too many applications running. Just realize that the SWAP is a failsafe for the RAM. It is definitely not as fast or efficient as RAM, but it's better than Window's Pagefile! (which accomplishes the same purpose)

EDIT: If you are interested in learning more about SWAP, see here.

Solution 2:

kswapd0 runs at 99.9% of one CPU but is actually not swapping at all

To me it happens at times on Ubuntu 14.04 with kernel 3.19.0-50-generic (and earlier) running in a VMware vm. I have no clue, what made it appear, but it comes during idle time.

top shows:

# top
top - 09:49:35 up 5 days, 18:35,  1 user,  load average: 1.00, 1.00, 0.99
Tasks: 219 total,   2 running, 217 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us, 25.0 sy,  0.0 ni, 74.7 id,  0.2 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem:   3028784 total,  1874468 used,  1154316 free,  1010276 buffers
KiB Swap: 15624188 total,     3032 used, 15621156 free.   234928 cached Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    52 root      20   0       0      0      0 R  99.7  0.0 122:15.21 kswapd0
     3 root      20   0       0      0      0 S   0.3  0.0   0:29.86 ksoftirqd/0
     7 root      20   0       0      0      0 S   0.3  0.0   9:49.47 rcu_sched

Temporary solution

a reboot solved the problem - temporarily.

following the answer on serverfault (kswapd often uses 100% CPU when swap is in use) there where the same settings on my system:

# cat /proc/sys/vm/swappiness
60
# cat /proc/sys/vm/vfs_cache_pressure
100
# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

the solution was actually running echo 1 > /proc/sys/vm/drop_caches as root user:

# cat /proc/sys/vm/drop_caches
0
# echo 1 > /proc/sys/vm/drop_caches
# cat /proc/sys/vm/drop_caches
1

Or, as pointed out by theTuxRacer (thanks!), use the following command in case you are not logged in as root:

echo 1 | sudo tee /proc/sys/vm/drop_caches

now it's fine:

# top
top - 10:08:58 up 5 days, 18:55,  1 user,  load average: 0.72, 0.95, 0.98
Tasks: 220 total,   1 running, 219 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.2 sy,  0.0 ni, 99.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   3028784 total,   681704 used,  2347080 free,     2916 buffers
KiB Swap: 15624188 total,     3032 used, 15621156 free.    81924 cached Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
     9 root      20   0       0      0      0 S   0.3  0.0  14:10.40 rcuos/0
     1 root      20   0   45652   8124   2888 S   0.0  0.3   1:54.98 init

Permanent solution (to be found)?

but since the actual reason is not yet known, and I did not fine any suitable explanation on the net, this is not a permanent solution. Actually, the selected answer could be the permanent solution. I just wanted to add this for future reference, as a reboot (to make sysctl take effect) is not always possible.

An other solution might be to set THP to either madvice or never (see poige's comment to his answer, How do I modify “/sys/kernel/mm/transparent_hugepage/enabled” and the referenced MongoDB Manual on Disable Transparent Huge Pages (THP))

cron job

i've set up the following batch as a cron job as a "permanent" solution:

#!/bin/bash
# Rev 2: Use ps instead of top

## run as cron, thus no $PATH, thus need to define all absolute paths
cpu=$(/usr/bin/printf %.0f $(/bin/ps -o pcpu= -C kswapd0))

[[ -n $cpu ]] \
&& (( $cpu >= 90 )) \
&& echo 1 > /proc/sys/vm/drop_caches \
&& echo "$$ $0: cache dropped (kswapd0 %CPU=$cpu)" >&2 \
&& exit 1

exit 0

invoked from root@localhost:~# crontab -e with

# m h  dom mon dow   command
  * *  *   *   *     /bin/bash /path/to/batch/drop_caches.sh >> /var/log/syslog 2>&1

Note: Above cron job script has been adapted to include Fredrik Erlandsson's suggestion: A much simpler and more efficient way to determine kswapd0's CPU usage. Thanks!

16.0414.04swap

Solution 3:

A more permanent solution is to increase the size of your swap file.

kswapd will take 100% CPU when searching for swap space that doesn't exist (if your swap partition is full). htop will show if your swap space is full at the top of the screen or you can use the free -h command to check used swap.

After using the following commands to increase the size of my swap file from 4GB to 8GB, kswapd now runs at 0% CPU with swappiness set to 60^*.

sudo swapoff /swapfile
sudo rm  /swapfile
sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Click here for a more detailed description of these commands

This assumes that you are already using a swap file (default Ubuntu setup) and that the /swapfile is referenced in /etc/fstab. Also, your swap file may be larger or smaller depending on how much RAM you have so adjust accordingly.

_{* It should be noted that I use a SSD so disk performance is not an issue. However, swap is almost always a good thing if you're running out of RAM.}