What happens when a computer runs out of RAM?

A little ago, out of curiosity, I wrote a Java program that takes up a lot of memory very fast. (It adds BufferedImages to an arraylist in a loop.)

I kept the Windows System Monitor open to observe the RAM usage as it ran.

When I started the program, the RAM usage climbed really fast, and as it reached about 100% of my 6GB of RAM, my computer became extremely slow. After about one minute, I managed to terminate the program and my computer slowly recovered speed. What caught my attention was that the memory usage first dropped slowly, then suddenly it went right down to almost 0MB before climbing back up to to idle usage at around 2-2.5GB.

I took a screenshot of the RAM usage in the System Monitor:

I was wondering if anyone could explain what happens as the computer runs out of memory and why it doesn’t go back to idle right after the program ends but rather about 60 seconds later? Also, why does the usage go all the way down to nothing instead of just idle usage?

When you start reaching the limit of main memory your operating system will start to swap out memory to disk to make room for more stuff (this is called paging). This is why your computer is getting really slow because the disk drive is waaaay slower than RAM, even if you're on a SSD, and it's also the reason your program can go on without any main memory left.

Idle memory is often swapped out first so that active programs can have as much as possible in RAM. This is the reason memory usage goes down below the usual idle state. The system will start to swap in memory pages again and the idle memory will fill up in RAM eventually.