Why does OS X use swap memory when memory pressure remains low?
Solution 1:
This is one of the all-too-few times where the answer is officially documented by Apple.
The short answer is that the system tries to keep a certain amount of memory free so it can quickly provide memory to an application when the application requests it. When there is not enough free memory, then memory that has not been "touched" in a while gets swapped out. Often this is memory that the application "leaked", which is to say allocated, used for a while (maybe just once), and will never use again, but forgot to tell the OS it was done with.
In Yosemite and other versions of OS X, SystemUIServer was notorious for leaking lots of memory. This would be a prime candidate for getting swapped out, because it was not needed by the app, but could not be reclaimed by the OS. Swapping this memory out to disk is all upside for performance; the only downside is a relatively tiny amount of disk space used.