Linux swapiness - Adjusting Kernel VM settings

Edit: Rewrote the answer so that it's shorter and clearer I hope :)

You really need to understand how the VM subsystem works as a whole to start tweaking the tunables or you may find that you're not getting the results that you expect. This article is a pretty good starting point on how these settings work together with a desktop slant.

So more to your question. Swappiness controls when the VM subsystem reclaims process table pages by unmapping and paging them out, aka swapping. This tunable works by telling the VM subsystem to look for pages to swap when the % of memory mapped to process page tables + swappiness value is > 100. So a setting of 60 will cause the system to start paging out stale pages from the process page table when it is using more than 40% of your system's memory. If you want to allow your programs to use more memory at the expense of cache you'll want to lower the swappiness value. You'll also want to have a look at /proc/sys/vm/min_free_kbytes and /proc/sys/vm/vfs_cache_pressure. As this will also control how much memory is kept in reserve and how aggressive the caching is. See that article I linked to for more information on the latter of those.