Unnecessary Swapping in Linux
My suggestion is setting your swappiness to 0.
Setting your swappiness to a "lower number" may reduce the "frequency" of using swap, but setting it to 0 will force the kernel down a different code path, forcing all memory allocation to first use up all available memory and cache before using swap space.
You can set swappiness to zero by issuing the following:
echo 0 > /proc/sys/vm/swappiness