Limit private memory usage per user

I'm using cgroups with the memory controller to set a memory limit for each user (using the memory.limit_in_bytes setting).

The problem is that this setting also accounts cache usage. Therefore, if the limit is 1GB, and the user merely downloads or copies a 1GB file, their processes get killed. What's worse, the cached pages remain in memory, so the user's "memory usage" remains close to 1GB even when they have zero processes running.

Naturally, this makes no sense. I only want to limit total private (non-anonymous) memory usage per user. How can I achieve that?

Alternatively, get the OOM killer to try dropping the user's cached pages before going off killing processes, which doesn't even free the cached pages.

Posting what I think might be a better answer.

My requirement is preventing one user's runaway process(es) from crashing the entire system.

Linux already has a feature for doing exactly this: the OOM killer.

The OOM killer runs when the system runs out of memory, and favors processes that consume a lot of RAM quickly. It is also less likely to kill long-running / system (superuser) processes.

The OOM killer can be further tuned by tweaking the /proc/<pid>/oom_score_adj file. The setting is inherited by child processes, so you only need to set it on each user's root process. (See Documentation/filesystems/proc.txt, section 3.1)