How do I escape from a hanging Ubuntu OS?

The least invasive option is to use Ctrl+Alt+F3 to switch to a console.

From the console, you can investigate what is going on using the tools you would normally use, like top, htop, free, or spelunking through the /proc and /sysfs file systems. Once your investigation has determined the culprit, you can send do whatever you need to correct the situation.

E.g. if you have a runaway process, you can try to send it the SIGTERM signal using kill -s TERM <PID>. If that doesn't work, you can be more aggressive and send it the SIGKILL signal using kill -s KILL <PID>.

If you have an SSH server set up, you can also log in remotely via SSH. Note, however, that if something on your system is going haywire, SSH might not work properly, e.g. because it can't get enough CPU time or memory or network sockets.

If you have a serial console set up, you can log in via serial terminal.

If you can't get to a console, the Linux Kernel has a feature called the "Magic SysRq Key".

The Magic SysRq Key is a way of getting the kernel's attention under almost all circumstances (the only major situation where it doesn't work is in a kernel panic, which makes sense: the kernel has crashed, it can't do any further work).

You can access the functionality of the Magic SysRq Key by holding down Alt+SysRq and typing a series of commands that are all single letters.

If you don't have a separate SysRq key, it is usually an alternate mode of the PrtScr key, even if it is not labelled. If you don't have that either, you'll have to look it up in the manual of your keyboard or your laptop.

Note that the keys are based on their scancodes on a US English QWERTY keyboard. In other words, Linux doesn't know what letter is printed on your key, it only cares about the location of the key on the keyboard. If you have a different keyboard layout, you will need to use the keys at the same position as if you had a US English QWERTY layout.

One common command sequence is:

• r: Switch the keyboard into XLATE mode from raw mode (e.g. if you are currently in a GUI, the keyboard is most likely switched to raw and will be useless in the console).
• e: Send the SIGTERM signal to all processes except PID 1 (init). This will nicely ask all processes to shut themselves down.
• … wait a bit to give all processes a chance to shut down orderly.
• i: Send the SIGKILL signal to all processes except PID 1 (init). This will very much not nicely ask all processes that are still running after the previous step to commit immediate suicide.
• … wait a bit
• s: sync all currently mounted file systems. This will flush the page cache and block cache to disk.
• … wait a bit to allow the Linux kernel to flush all caches to the storage devices and the storage devices to flush all their internal caches to the medium; if you still have old-style disk activity LEDs, wait until activity dies down
• u: umount all currently mounted file systems, or rather remount all currently mounted filesystem read-only.
• … wait a bit
• b: reboot or alternatively o: shutdown

The sequence could be written out as REISUB. A common mnemonic to remember it is "Reboot Even If System Utterly Broken".

You can set a shortcut to open system monitor in Ubuntu with the key combination you like (in this case, Ctrl+Alt+Del).

You can set custom shortcuts for custom key combination in the Settings (by the way, these steps are for Ubuntu 18.04):

1. Go to the Settings.
2. Search for shortcuts (just start typing and it will search for it).
3. Now, press on the Keyboard option and scroll down to the end.
4. Click on the + sign and a dialog will come up.
5. Type the name of the shortcut which can be anything you want (in this case, you can write Open System Monitor).
6. Type the name of the command (in this case, gnome-system-monitor).
7. Click on the Set shortcut` option and press the key combination you will use to open the System monitor (in this case, press Ctrl, Alt and Del at the same time after clicking the option).
8. Last but not the least, click Add and a custom shortcut will be created.

You can try out the shortcut by pressing the key combination(Ctrl+Alt+Del) you assigned to the shortcut.

Holding down Alt and SysRq (which is the Print Screen key) while slowly typing "REISUB" will get you safely restarted. "REISUO" will do a shutdown rather than a restart.

If you know the program that is not responding, go to a TTY shell CTRL+ALT+F4 then use the following script to kill that program. Copy the code and save it as exterminate.sh in /bin

#!/bin/bash

userName=$(whoami);
programName="$1";
main(){
    pidVar=$(top -n 1 -d 1 -b | grep -i "$programName" | cut -c -6 );
    kill $pidVar >& /dev/null;
    echo "killed all \"$programName\" instances.";
}

if [ "$userName" == "root" ] ; then main ; else echo "run as root" ; fi;

then make the script executable by

sudo chmod +x /bin/exterminate.sh

and execute it as superuser. Be careful about the argument that you pass to the script as you inadvertently might end up causing more damage than you've bargained for.

e.g. if for example the malfunctioning program is transmission

sudo exterminate.sh transmission

would do the trick.

when my PC overloads, my Ubuntu system hangs

For me, this usually means running out of RAM and starting to swap heavily.

What can help in this situation is the Alt+SysRq+F combination, which commands the system to kill the task taking most memory.

You'll have to enable the SysRq key first, as explained in this question. Also some laptops require holding down Fn key simultaneously.