What's using my swap (Ubuntu)?

Solution 1:

The swap usage patterns you describe don't sound surprising. They're consistent with some permanently-running processes having rarely-used pages. During the day, due to the high activity, the rarely-used pages are almost always in the swap. At night, there's more room for them in RAM.

You can get a glimpse of how much memory of various kinds each process is using in top or htop. Neither show swap usage by default, but both can be configured to (top: press f and switch on the SWAP column; htop: press F2, add the NSWAP column). You can get more information about a particular process with cat /proc/12345/vmstat where 12345 is the process ID. Note that “how much swap a program is using” is not completely well-defined, as some pages are shared by several processes.

There are two major kinds of competitors for RAM: process memory (which can be swapped out) and disk caches (which can be re-read from a file). There is no reason to always prioritize process memory over disk caches: it's better to swap out a rarely-used portion of process memory than keep reading a file into memory again and again. The figures you give, with about half (say 30%–70%) of the memory devoted to disk caches, is typical for systems that have a reasonable amount of RAM for the tasks they're supposed to do.

Solution 2:

On StackOverflow, there is already an excellent answer on your first question ("Which process is actually using swap?").

https://stackoverflow.com/a/7180078/1442301

In a nutshell, tools like top or htop does not provide accurate information at all about swap usage. You should dive into the /proc folder (in the pseudo file /proc/$PID/smaps) to get more accurate information.

The post mentioned above contains a short shell script to get this information echoed in a nice way.