free reports bogus amount of swap used
The swap used reported by free is very high.
[root@rhel6 ~]# free -m
total used free shared buffers cached
Mem: 9892 9537 354 0 71 884
-/+ buffers/cache: 8581 1310
Swap: 767 1759218592 116869
Like, really high.
[root@bb14 blackboard]# free -g|grep Swap
Swap: 0 1717986906 114
Or is it?
[root@bb14 blackboard]# free -h |grep Swap
Swap: 767M 767M 114G
Even more strange, even if I disable swap the number still stays high.
[root@rhel6 ~]# swapoff -a
[root@rhel6 ~]# free -m
total used free shared buffers cached
Mem: 9892 9760 131 0 45 638
-/+ buffers/cache: 9076 815
Swap: 0 1759218592 116814
Things don't get any less confusing when checking meminfo, which shows swapfree higher than swaptotal.
[root@rhel6 ~]# cat /proc/meminfo|grep Swap
SwapCached: 0 kB
SwapTotal: 786428 kB
SwapFree: 120404008 kB
Obviously something is wonky and my first instinct is to reboot, but this is a production machine which means maintenance windows, etc, and I'm wondering if there's any way to find out what's wrong and possibly even fix it without downtime.
The solution was to Upgrade to kernel-2.6.32-573.7.1.el6 or higher.
A simple yum update
and reboot should be all it takes.
Below is a quote of the bug report of kernel-2.6.32-573.1.1.el6.x86_64 that caused the Swap free to be greater than the swap total:
A previous change in the get_swap_page() locking removed the use of the swap_lock spinlock. This could cause nr_swap_pages corruption and invalid SwapFree information in the /proc/meminfo file, where the size of SwapFree could exceed the size of SwapTotal. This update uses an atomic variable for nr_swap_pages, and the size of SwapFree in /proc/meminfo is now correct. (BZ#1259362)
It looks like this is a known issue in RHEL6.7 with kernel-2.6.32-573.1.1.el6.x86_64 being tracked in their private bugzilla.
https://access.redhat.com/solutions/1571043