Why does host swap out VMs when there are 16GB of buffer cache and swappiness = 0? [duplicate]
I have a desktop running Ubuntu Quantal using OpenStack Folsom on an Intel i5 with 32 GB RAM and 2 GB swap. I'm running 7 VMs each sized like a EC2 m1.small, so 1.7 GB RAM each. I'm using KVM.
As I get up to running 5 or 6 concurrently, the host starts to swap them out:
top - 23:45:42 up 3 days, 1:51, 10 users, load average: 0.37, 0.75, 1.15
Tasks: 418 total, 2 running, 413 sleeping, 3 stopped, 0 zombie
%Cpu(s): 8.8 us, 2.1 sy, 0.0 ni, 88.8 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem: 32864580 total, 32586956 used, 277624 free, 574236 buffers
KiB Swap: 1998844 total, 1113352 used, 885492 free, 16498252 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24652 libvirt- 20 0 4169m 1.7g 7756 S 3.6 5.4 4:49.37 kvm
25233 libvirt- 20 0 4450m 1.6g 7756 S 1.2 5.2 4:35.12 kvm
25589 libvirt- 20 0 4163m 1.6g 7756 S 2.4 5.1 4:40.31 kvm
6562 root 39 19 2935m 658m 7460 S 0.0 2.1 100:05.62 java
28393 libvirt- 20 0 4149m 624m 7756 S 0.0 1.9 2:25.01 kvm
28106 libvirt- 20 0 4170m 617m 7756 S 0.0 1.9 2:18.17 kvm
26519 libvirt- 20 0 4167m 590m 7756 S 0.0 1.8 2:22.16 kvm
29399 libvirt- 20 0 4159m 589m 7756 S 0.0 1.8 2:19.94 kvm
$ free -m
total used free shared buffers cached
Mem: 32094 31868 225 0 559 16175
-/+ buffers/cache: 15134 16959
Swap: 1951 1087 864
# /tmp/swap-used.sh |grep kvm
PID=944 - Swap used: 0 - (kvm-irqfd-clean )
PID=24652 - Swap used: 102468 - (kvm )
PID=25233 - Swap used: 108644 - (kvm )
PID=25589 - Swap used: 155768 - (kvm )
PID=26519 - Swap used: 192216 - (kvm )
PID=28106 - Swap used: 150796 - (kvm )
PID=28393 - Swap used: 208488 - (kvm )
PID=29399 - Swap used: 187388 - (kvm )
I've already tried setting swapiness to 20, then 10 and finally to 0, none of which has made a difference:
# cat /proc/sys/vm/swappiness
0
I haven't rebooted the host since changing it from 60 to 0 (do I need to reboot)? I've also turned off swap completely with /sbin/swapoff -a; /bin/swapon -a. Immediately after re-enabling swap, I see this:
$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 11384 247572 652736 16539748 11 10 291 228 13 12 7 3 87 3
0 0 12968 234360 652756 16554432 0 317 8576 1240 3508 5470 17 2 75 5
1 0 17068 243512 652768 16559508 0 820 9448 1216 3687 4845 20 2 77 2
1 0 20040 233300 652772 16576152 0 594 12262 677 4436 5063 29 2 68 1
1 0 22764 219156 652792 16594448 6 546 11962 727 3870 4559 28 1 68 2
3 0 40832 229384 652776 16602440 0 3614 58404 4176 2051 6231 21 2 66 10
1 0 52420 232236 652784 16613320 0 2318 42174 2512 1819 4026 15 2 77 6
I've got 15 GB of free memory that can be used without having the processes swapped out.
I've got 15 GB of free memory that can be used without having the processes swapped out.
No, you don't.
total used free shared buffers cached
Mem: 32094 31868 225 0 559 16175
You have 225MB of memory free.
KiB Swap: 1998844 total, 1113352 used, 885492 free, 16498252 cached
Look at all that wonderful cache the system has made, in part by getting unused junk out of precious RAM. Why are you trying to make it waste precious RAM?
If you have 1GB of data in RAM that hasn't been used in hours, which makes more sense:
You swap that out to disk when you're not busy and have 1GB more cache to use.
You keep it in memory. You have 1GB less cache to use, and if that memory is ever needed for something else, you'll have to swap it out while you are busy.
But if you think you know more about memory management than the people who wrote your operating system's memory management logic, go ahead and keep trying to find that magic "do everything faster and better" switch.