Prevent Ubuntu from freezing, even if system memory is low

Sometimes I work with huge dumps of data I want to keep in memory for processing. Sometimes I miscalculate the amount of memory my program will produce, or a debugger multiplies the memory usage by a factor that exceeds my available memory.

Whenever I start a memory-hungry process, this is what I'd expect from a sane operating system: try to eat all free memory, then ask some other non-essential processes nicely to give up some memory they don't need, then write to swap.

Here's what Ubuntu does for me: eat all fre memory, then ask the operating system to swap all essential services (gnome session, terminal, keyboard), then freeze and wait for me to pull the power plug.

Two questions:

1. How can an operating system assume, that anything could be so important that it is ok to stop listening to user input?
2. How can I tell Ubuntu to never swap essential services and always react to user input, even if some stupid process tries to eat up more resources than the system provides.

Solution 1:

I still don't have a solution for the problem, but I can offer two workarounds which might be of interest to others:

1) earlyoom

That's a service that watches memory usage and kills the process that consumes the most memory when a certain threshold is reached (see also this and this question regarding the OOM killer in the linux kernel)

I've tested it with a demo process that indefinetly requests memory in small chunks. Here's my first impression: When I start the rogue process, it quickly eats up all of my RAM. Then swapping begins, the system becomes inresponsive. A few seconds later the system is back online. The log of earlyoom shows that it killed the memory eating process after both memory and swap usage reached 90%.

There is still the annoying lag when swapping begins and after the process was killed, some portions of other processes usually remain in swap until they are requested, but it's a start.

2) just disable swap

I know this is a controversial topic, but for the purpose of desktop systems and especially development machines where it can happen from time to time that a process tries to eat up all your memory, it makes sense: Without swap, the OOM killer just works as intended. When you run out of memory, it finds the best process to kill and gets rid of it. No lag, no delay.

You can disable swap for the current session with sudo swapoff -a or make the change permanent.

The proper solution for the problem would of course be that the system stays responsive when the main memory is depleted and it starts to swap memory like there's no tomorrow, but that doesn't seem to be happening any time soon.