Swap with a huge amount of ram available
I have an old, legacy server with an odd problem with swap.
- Linux version: Red Hat Enterprise Linux Server release 5.6 (Tikanga)
- Kernel version: 2.6.18-238.el5
- Server is virtual.
- Server has 2 virtual socket.
I know swap partition is to small, going to add a swap file, but, after few hours after reboot, the situation is this:
free -m
total used free shared buffers cached
Mem: 15922 15806 116 0 313 13345
-/+ buffers/cache: 2147 13775
Swap: 2047 2042 4
Oracle database is installed, but almost unused. I'd like to understand why memory distribution goes this way. I mean 13345 cached, means free. Why filling swap?
A previous sysadmin configured swappiness to: 3.
Huge pages are not configured.
I saw some post similar, but with no solution to understand. An answer here: linux redhat 5.4 - swap while memory is still available talks about numa, so I digged a bit (I'm a dba, not a sysadmin, so sorry if I miss something).
grep NUMA=y /boot/config-`uname -r`
CONFIG_NUMA=y
CONFIG_K8_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_ACPI_NUMA=y
dmesg | grep -i numa
NUMA: Using 63 for the hash shift.
So, the question is: how can I understand why is this machine swapping?
Update With a: vmstat 2
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
4 0 2090852 122224 324056 13679328 320 0 498 1898 1088 3555 32 10 56 2 0
1 0 2090724 139740 324068 13680984 64 0 76 932 1028 3534 7 2 90 2 0
0 0 2090724 132416 324068 13681436 0 0 16 240 1016 3401 3 1 96 1 0
4 0 2090660 116916 324084 13683404 0 0 72 1396 1070 3617 11 9 80 1 0
0 0 2090420 126544 324084 13687008 128 0 188 1872 1068 3436 35 8 56 2 0
Update 3
ipcs -ma
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x61a4d538 5210113 oracle 660 4096 0
0xba8cafdc 5242883 oracle 660 4096 0
0x16621634 5308420 oracle 660 4096 0
0xc15f3dac 5373957 oracle 660 4096 0
------ Semaphore Arrays --------
key semid owner perms nsems
0x24690d60 98304 oracle 660 125
0x24690d61 131073 oracle 660 125
0x24690d62 163842 oracle 660 125
0x24690d63 196611 oracle 660 125
0x24690d64 229380 oracle 660 125
0x24690d65 262149 oracle 660 125
0x24690d66 294918 oracle 660 125
0x24690d67 327687 oracle 660 125
0x24690d68 360456 oracle 660 125
0x6285541c 491529 oracle 660 125
0x6285541d 524298 oracle 660 125
0x6285541e 557067 oracle 660 125
0x6285541f 589836 oracle 660 125
0x62855420 622605 oracle 660 125
0x62855421 655374 oracle 660 125
0x62855422 688143 oracle 660 125
0x62855423 720912 oracle 660 125
0x62855424 753681 oracle 660 125
0xaee7ccbc 884754 oracle 660 125
0xaee7ccbd 917523 oracle 660 125
0xaee7ccbe 950292 oracle 660 125
0xaee7ccbf 983061 oracle 660 125
0xaee7ccc0 1015830 oracle 660 125
0xaee7ccc1 1048599 oracle 660 125
0xaee7ccc2 1081368 oracle 660 125
0xaee7ccc3 1114137 oracle 660 125
0xaee7ccc4 1146906 oracle 660 125
0xfb4a455c 1277979 oracle 660 125
0xfb4a455d 1310748 oracle 660 125
0xfb4a455e 1343517 oracle 660 125
0xfb4a455f 1376286 oracle 660 125
0xfb4a4560 1409055 oracle 660 125
0xfb4a4561 1441824 oracle 660 125
0xfb4a4562 1474593 oracle 660 125
0xfb4a4563 1507362 oracle 660 125
0xfb4a4564 1540131 oracle 660 125
------ Message Queues --------
key msqid owner perms used-bytes messages
If you aren't experiencing lots of active paging (si/so in vmstat) then it's nothing to worry about. The kernel is opting to put bits of program code that is not actively being used out to swap so that it can keep more of the DB files in RAM (cached in free).
This is an excellent artice I've found about swap and how having it used isn't always bad. https://chrisdown.name/2018/01/02/in-defence-of-swap.html
Not good. If you have anything except a very incidental si you are basically outside of the realm of "good swap". And your system is doing it all the time. SwapIn means that some program is waiting and it potentially means that user experiences a slow down (local or remote user).
I'm all about that good swap. Which means mostly so to push out the bloated stuff out of your precious RAM; since it's bloated it sits on disk not used by anyone with a very very very incidental swapin activity.
You've hit a funny thing with Oracle that I saw before: somehow Oracle reports its SGA as a "cache" (if I remember correctly, because it's an anonymous mmap). But being anonymous, it's not a memory that a system can simply drop at any time, thus it's not free! Quite the opposite - it's heavily used. Oracle often uses also actual remaining system cache when reading data files which can put a lot of memory pressure on the cache (dumb default behavior).
So it's one of many cases when the free misleads you to think that all "cache" is to be treated as "free". That rule of thumb is only for a mostly-read-only file server. (I think that's why authors of free have put the line that says "cache" is "used" above the line where it implies "cache" is "free".)
And don't tune swappiness at all as long as a system has 100% occupied swap. That's... not wise.
I bet your SGA isn't 2 GB, but maybe more like 10 GB or even more. I think that if you configure the SGA to 6 GB you will see that swapins will decrease and much less stuff will be pushed to swap. As you increase it step by step, you will see how the pressure rises.