How to prevent my system from becoming unresponsive when thrashing (out of RAM)?

I have 8GB of RAM, but sometimes I'm running a couple VMs and enough Chrome processes to use up most of my memory. That's usually fine, but sometimes my system starts thrashing, and it's difficult for me to recover from it without just waiting for things to get better since the UI is mostly frozen.

In all of these cases, the programs using most of my RAM could be deprioritized. If Chrome or the VM became nonresponsive but my graphical environment and terminal were still responsive, I could switch over to a terminal, find the culprit, and kill it. I could imagine something like a "nice" for memory (so that I could nice Chrome and my VM) or something to force the graphical environment and my terminal to stay in memory and not to swap.

Is there a way to ensure that the UI remains responsive when my system is thrashing?

Thanks!

EDIT: I'm still interested in whether there's something like a nice for memory, but I just discovered that my sistem doesn't have any swap space. I think that this happened because I started using an encrypted home and encrypted swap (see Swap not working on clean 14.04 install using encrypted home). So, my system is probably becoming unresponsive because I don't have any swap space, so my system can't swap out unnecessary programs when it gets low on memory.

I just discovered that my sistem doesn't have any swap space

If this is true, then it's probably contributing to the problem.

By using multiple VMs with large memory allocations, an amount of RAM that would normally be more than enough for normal use (8GB) starts becoming tight, which means your system won't be able to use as much cache, and will want to swap. Now, if you don't have swap, there will be even more pressure on memory and you may even get tasks killed due to running out of memory.

Swap tries to improve overall stability and performance in situations like the one you have described, by re-allocating some allocated but seldom-used data in memory to disk, freeing memory for data you are using. Adding swap, if you indeed have none, will help. However, in performance terms you'll make much more improvement if you also buy more RAM, or use less of it.

On a system where you're running VMs but you're also using a regular desktop on the VM host, you don't want your VMs to take up all your RAM, in fact I'd save half your RAM (up to 4GB) for the host, given you're running software on the host and want it to be responsive.

Note too that VMs have inherent inefficiencies not just because their RAM is totally separate they don't share any cache or buffers, but they are running a whole separate OS under a number of layers of abstraction. So you can always get better efficiency if you decide to run something on the host rather than in a VM - though if your purpose for using VMs is to use other operating systems this won't be possible.

Is there a way to ensure that the UI remains responsive when my system is thrashing?

When the amount of RAM you have is a problem, anything you can do is only a hack - the real solution is to buy more RAM (although in your case as I mentioned you may be able to just allocate less to the VMs).

Your question is put in simple terms - you want the UI to be able to continue to operate smoothly even if IO is congested. However, it's complicated to answer. Your system will already be doing this to the best of its ability: operations that don't require disk access won't be being delayed. If your "UI is stuttering" it's because the particular process responsible for that UI needs disk access. In a low memory situation and where the disk is thrashing, even something which will normally be trivial - such as reading some config from your home directory to find out what your display preferences are - can be slow due to depletion of disk cache.

I could imagine something like a "nice" for memory (so that I could nice Chrome and my VM)

There isn't really a "nice for memory". Memory access is so fast that you can usually practically ignore it. When you have thrashing, it's not memory, it's disk, that's the problem. There is an "ionice" because of this. Though that's probably not what I'd recommend for your issue.

Another rough equivalent of "nice for memory" is swap - you can't control how swappy each process is individually but the system does a pretty good job of working out which processes you are actively using and therefore are a higher priority to be kept in RAM.

But as I said and as is a golden rule, if you're running out of RAM, then buying more RAM is the solution with the least headaches and drawbacks.