Constantly Increasing swap size in Linux and Swap space not being reclaimed?

I have a 8GB RAM linux box on which 4 tomcat servers are running. One of the them is set to 3000MB memory(jvm -Xms and -Xmx setting) and others are set to 1500MB. The swap partition is also set to 8Gigs. When I start these servers, the swap file usage is low. But over a period of days and during certain times when one/all of the servers are in peak activity , the swap usage starts increasing. Here's a typical sar -r output.

kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad

48260 8125832 99.41 196440 2761852 7197688 1190912 14.20 316044

75504 8098588 99.08 198032 2399460 7197688 1190912 14.20 316032

It shows 14.2% swap used currently. The funny thing is this % NEVER decreases. It continues to increase and reach up to 30-40%. We restart our servers weekly.

I would assume the %swpused to increase during periods of peak activity and decrease during periods of low activity..Or at least remain constant. This looks like the swap space is never reclaimed by the OS..

Output of free : free -m total used free shared buffers cached Mem: 7982 7937 45 0 32 2088 -/+ buffers/cache: 5816 2166 Swap: 8191 1163 7028

So there's at least 2g of free Ram. So the question is Why is the swap space continuing to increase and not being reclaimed by the OS ? Or how to debug this to figure out whats hapenning..

If info is swapped out to disc and later read back into memory, it will often be left allocated in the swap area until swap space runs low. That means that if the same info needs to be swapped out again later and hasn't changed, the OS can just drop the pages from allocated RAM without needing to write anything to disc saving time.

Swap allocated to stuff that has been read back into memory will be freed either

1. when the relevant pages are no longer needed at all (i.e. are freed by the application)
2. when the relevant pages are changed (so the copy on disc is no longer up to date)
3. the machine runs low on swap space so clears some things that are already in RAM to make room

Look in /proc/meminfo for a line called "SwapCached". This entry counts pages that are found both in RAM and in swap partitions. For instance, picking a small VM at random, the /proc/meminfo virtual file one of my VMs shows:

SwapTotal:        698816 kB
SwapFree:         624520 kB
SwapCached:        17232 kB

indicating that 74268K of swap space is allocated, but that 17232K worth of those pages are also currently mapped into RAM too (so could be deallocated from swap at a moment's notice if the space is needed by something else).

Also there will no doubt be pages sat there that was swapped out ages ago and have never been used again since. The kernel will not reload pages from swap just because there is some free RAM to read it back into as that free RAM might be better used for cache or buffers - pages written to swap are generally only reread when they are next needed.

If you want to clear out what is in swap, as long as you have enough free and/or freeable (i.e. free+cache+buffers (less those parts of the c+b counts that are not freeable RightThisInstant)), just turn it off and back on again with swapoff -a && swapon -a.

Of course you could also have a memory leak somewhere too, but that is not the only explanation for the behaviour you are seeing.

Basically, you dont need to care about this. What's important to know is how much IO goes to the swap (have a look at the 'vmstat' command). Having more stuff in the swap doesnt cost anything. The only cost is putting stuff in the swap (page in) or taking it out (page out). So it is perfectly reasonalbe for the OS to let the swap grow.