How can I prevent my system from crashing or becoming unusable due to high RAM usage?

Sometimes when I use Lubuntu 17.10 - fully updated, my RAM usage hits 100% and the system becomes unusable, if I'm "lucky" I can move my mouse about 1 pixel per 10 seconds... This makes it practically impossible to do anything even shutdown an applications. I am then forced to use the power button on my laptop.

I'm using a macbook pro retina late 2013, with almost fresh Lubuntu 17.10 install.

So far it has happened while I have spyder3 open and I've loaded too big files. It also happened when I clicked this link in firefox (WARNING: might crash) http://stats.oecd.org/restsdmx/sdmx.ashx/GetData/SNA_TABLE1. I even closed the tab before hitting 100% usage, but it just continued up to 100% and crashed. I only noticed because I closed the tab and saw RAM usage in spyder3 (I was not running any python3 scripts).

Is there a way to prevent this? Perhaps make sure a specific amount of RAM can only be used by the OS?

If a process eats too much memory then the system should protect itself through the OOM killer, wich is a standard feature in any Linux system.

As estated in the best answer so far to How does the OOM killer decide which process to kill first?

If memory is exhaustively used up by processes, to the extent which can possibly threaten the stability of the system, then the OOM killer comes into the picture.

NOTE: It is the task of the OOM Killer to continue killing processes until enough memory is freed for the smooth functioning of the rest of the process that the Kernel is attempting to run.

The OOM Killer has to select the best process(es) to kill. Best here refers to that process which will free up the maximum memory upon killing and is also the least important to the system.

The primary goal is to kill the least number of processes that minimizes the damage done and at the same time maximizing the amount of memory freed.

To facilitate this, the kernel maintains an oom_score for each of the processes. You can see the oom_score of each of the processes in the /proc filesystem under the pid directory.

$ cat /proc/10292/oom_score

The higher the value of oom_score of any process, the higher is its likelihood of getting killed by the OOM Killer in an out-of-memory situation.

If your system crashes then I'd suggest to fine tune the OOM policy by adjusting the oom_score of your processes.

It is unlikely that the OOM killer was disabled but, to make sure, check if this command returns 0:

$ sudo sysctl vm.overcommit_memory
vm.overcommit_memory = 0

References:

1. http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html
2. https://superuser.com/questions/1150215/disabling-oom-killer-on-ubuntu-14-04/1150229
3. https://unix.stackexchange.com/questions/153585/how-does-the-oom-killer-decide-which-process-to-kill-first
4. https://www.kernel.org/doc/Documentation/sysctl/vm.txt

Probably your problem is caused by the system "thrashing" – moving many pages of memory to and from swap space at once, and not leaving time for real processes to execute.

If you want processes that use too much memory to be killed rather than making your whole system run slowly, you can disable swap. Running sudo swapoff -a will achieve this until a reboot; to disable swap permanently, you need to edit /etc/fstab to remove the swapfile/swap partition, by removing or commenting out the line with swap in the third column. For example, my /etc/fstab looks a bit like this:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/mapper/ubuntu--vg-root /               ext4    errors=remount-ro 0       1
# /boot was on /dev/sda1 during installation
UUID=fdedeca9-dafe-1a05-0866-7502fda1a7ea /boot           ext2    defaults        0       2
/dev/mapper/ubuntu--vg-swap_1 none            swap    sw              0       0

To disable swap, I would comment out the last line (by putting a hash # at the beginning of the line), then reboot. Make sure you don't modify any of the rest of the file, otherwise you might make your system unable to boot.

Caveat: If you disable swap and don't have enough physical memory for basic system services, the OOM killer may decide to kill one of those and lead the system to crash or become otherwise unusable (which the question says you want to avoid).