OS X: Memory usage freaking me out

The values in your image look fine. Since the system is lagging, why not measure things over time:

vm_stat 60

You can let that run in a terminal to get an aggregate feel for when swapping happens and at what rate. It's highly likely that some other thing is the bottleneck like CPU or storage, but without seeing things change over time, it's hard to say what's causing the lag. I've never seen a system lag due to RAM without the memory pressure being yellow or worse, but that doesn't mean it's impossible to have.

That coupled with the fact that your system still has 10 GB of free RAM makes it even less likely it's contention for memory/swap causing slowness. Feel free to ping me if I miss an edit to the question with more data to go on.