how to configure Linux to call oom_kill on its own

Solution 1:

OOM doesn't necessarily kill the process you think it will. Read about it here: http://lwn.net/Articles/317814/

I suspect what is happening is that because chrome uses a different process for each page, it's confusing the OOM-killer's heuristics and other processes are being killed rather than the parent chrome process.

According to this, the OOM-killer likes to kill niced processes. So try setting chrome to a niceness of greater than 0.. might make it more of a target :-)

Solution 2:

I have come to the conclusion that modern linux doesn't like to run without virtual memory. Maybe getting rid of swap made sense in the days of slow hard drives / older VM algorithms / etc. Or maybe you could jump through some hoops to configure the oom killer...

But I've been running with a swap file for the past few months and I'm much happier with my machine's low-memory behavior. It slows down a bit at the limits, but it's easy to intervene and kill memory-hungry processes to get things back on track.