Determining the source of memory cache usage

To find out about "memory cache" use slabtop, using -s you can sort the output and c is for the cache size, so use:

sudo slabtop -s c

For me most of the cache is related to inode_cache


And about "swap", You can use status file in each process directory in /proc to find out which one of them is using the swap.

For specific program:

cd /proc/$(pgrep -x programname)
grep -i swap status

To get a list of all process cache size:

cd /proc
find -maxdepth 2 -iname status -exec grep -i -e name -e swap {} \; -exec echo "---" \;

the output would be similar to:

---
Name:   atd
VmSwap:        0 kB
---
Name:   rsyslogd
VmSwap:        0 kB
---
Name:   cron
VmSwap:        0 kB