Is there a way to reserve memory for OS to keep it active when RAM usage goes to 100%?

Ubuntu 18.04

I keep having same issue - due to my bad habbits, my Firefox gets to open and hold in memory too many tabs. This causes RAM usage to eventually hit 100%
At which point whole system hardlocks. I can't even move my cursor to press "end process" in system monitor.

Same behavior appears when I run my code and take too much memory - everything stops working, mouse becomes unresponsive.

My guess is obvious - shortage of free memory blocks Gnome or Ubuntu itself.
So my guessed solution would be to make those OS systems reserve memory to always work and respond.

How do I do that?

as requested

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           7,7G        4,3G        281M        1,2G        3,1G        1,9G
Swap:          2,0G        1,5M        2,0G

$ sudo sysctl vm.swappiness

vm.swappiness = 60

Solution 1:

Instead of a thousand words: system hang prevention with nohang: https://youtu.be/PLVWgNrVNlc

Nohang is a highly configurable daemon for Linux which is able to correctly prevent out of memory (OOM) and keep system responsiveness in low memory conditions: https://github.com/hakavlad/nohang

But easiest way is installing earlyoom. This program is in the repositories starting with ubuntu 18.04: just run sudo apt install earlyoom.