Swap with a huge amount of ram available

I have an old, legacy server with an odd problem with swap.

  • Linux version: Red Hat Enterprise Linux Server release 5.6 (Tikanga)
  • Kernel version: 2.6.18-238.el5
  • Server is virtual.
  • Server has 2 virtual socket.

I know swap partition is to small, going to add a swap file, but, after few hours after reboot, the situation is this:

free -m
             total       used       free     shared    buffers     cached
Mem:         15922      15806        116          0        313      13345
-/+ buffers/cache:       2147      13775
Swap:         2047       2042          4

Oracle database is installed, but almost unused. I'd like to understand why memory distribution goes this way. I mean 13345 cached, means free. Why filling swap?

A previous sysadmin configured swappiness to: 3.
Huge pages are not configured.

I saw some post similar, but with no solution to understand. An answer here: linux redhat 5.4 - swap while memory is still available talks about numa, so I digged a bit (I'm a dba, not a sysadmin, so sorry if I miss something).

grep NUMA=y /boot/config-`uname -r`
CONFIG_NUMA=y
CONFIG_K8_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_ACPI_NUMA=y

dmesg | grep -i numa
NUMA: Using 63 for the hash shift.

So, the question is: how can I understand why is this machine swapping?

Update With a: vmstat 2

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 4  0 2090852 122224 324056 13679328  320    0   498  1898 1088 3555 32 10 56  2  0
 1  0 2090724 139740 324068 13680984   64    0    76   932 1028 3534  7  2 90  2  0
 0  0 2090724 132416 324068 13681436    0    0    16   240 1016 3401  3  1 96  1  0
 4  0 2090660 116916 324084 13683404    0    0    72  1396 1070 3617 11  9 80  1  0
 0  0 2090420 126544 324084 13687008  128    0   188  1872 1068 3436 35  8 56  2  0

Update 3

ipcs -ma

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x61a4d538 5210113    oracle    660        4096       0
0xba8cafdc 5242883    oracle    660        4096       0
0x16621634 5308420    oracle    660        4096       0
0xc15f3dac 5373957    oracle    660        4096       0

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x24690d60 98304      oracle    660        125
0x24690d61 131073     oracle    660        125
0x24690d62 163842     oracle    660        125
0x24690d63 196611     oracle    660        125
0x24690d64 229380     oracle    660        125
0x24690d65 262149     oracle    660        125
0x24690d66 294918     oracle    660        125
0x24690d67 327687     oracle    660        125
0x24690d68 360456     oracle    660        125
0x6285541c 491529     oracle    660        125
0x6285541d 524298     oracle    660        125
0x6285541e 557067     oracle    660        125
0x6285541f 589836     oracle    660        125
0x62855420 622605     oracle    660        125
0x62855421 655374     oracle    660        125
0x62855422 688143     oracle    660        125
0x62855423 720912     oracle    660        125
0x62855424 753681     oracle    660        125
0xaee7ccbc 884754     oracle    660        125
0xaee7ccbd 917523     oracle    660        125
0xaee7ccbe 950292     oracle    660        125
0xaee7ccbf 983061     oracle    660        125
0xaee7ccc0 1015830    oracle    660        125
0xaee7ccc1 1048599    oracle    660        125
0xaee7ccc2 1081368    oracle    660        125
0xaee7ccc3 1114137    oracle    660        125
0xaee7ccc4 1146906    oracle    660        125
0xfb4a455c 1277979    oracle    660        125
0xfb4a455d 1310748    oracle    660        125
0xfb4a455e 1343517    oracle    660        125
0xfb4a455f 1376286    oracle    660        125
0xfb4a4560 1409055    oracle    660        125
0xfb4a4561 1441824    oracle    660        125
0xfb4a4562 1474593    oracle    660        125
0xfb4a4563 1507362    oracle    660        125
0xfb4a4564 1540131    oracle    660        125

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

If you aren't experiencing lots of active paging (si/so in vmstat) then it's nothing to worry about. The kernel is opting to put bits of program code that is not actively being used out to swap so that it can keep more of the DB files in RAM (cached in free).

This is an excellent artice I've found about swap and how having it used isn't always bad. https://chrisdown.name/2018/01/02/in-defence-of-swap.html


Not good. If you have anything except a very incidental si you are basically outside of the realm of "good swap". And your system is doing it all the time. SwapIn means that some program is waiting and it potentially means that user experiences a slow down (local or remote user).

I'm all about that good swap. Which means mostly so to push out the bloated stuff out of your precious RAM; since it's bloated it sits on disk not used by anyone with a very very very incidental swapin activity.

You've hit a funny thing with Oracle that I saw before: somehow Oracle reports its SGA as a "cache" (if I remember correctly, because it's an anonymous mmap). But being anonymous, it's not a memory that a system can simply drop at any time, thus it's not free! Quite the opposite - it's heavily used. Oracle often uses also actual remaining system cache when reading data files which can put a lot of memory pressure on the cache (dumb default behavior).

So it's one of many cases when the free misleads you to think that all "cache" is to be treated as "free". That rule of thumb is only for a mostly-read-only file server. (I think that's why authors of free have put the line that says "cache" is "used" above the line where it implies "cache" is "free".)

And don't tune swappiness at all as long as a system has 100% occupied swap. That's... not wise.

I bet your SGA isn't 2 GB, but maybe more like 10 GB or even more. I think that if you configure the SGA to 6 GB you will see that swapins will decrease and much less stuff will be pushed to swap. As you increase it step by step, you will see how the pressure rises.