What happens when linux OS out of RAM and no swap

When the operating system is out of RAM and has no swap, it discards clean pages. It cannot discard dirty pages because it would have to write them somewhere first. This causes thrashing and poor performance if there is insufficient RAM to hold the working set. That's one of the main reasons you really want swap -- so the operating system can make a better decision about what pages to evict.

With no swap, the system will run out of virtual memory (strictly speaking, RAM+swap) as soon as it has no more clean pages to evict. Then it will have to kill processes.

Running out of RAM is completely normal. It's just a negative spin on using RAM. Not running out of RAM could equally well be described as "wasting RAM". Once all RAM is in use, the operating system makes intelligent decisions about what to keep in RAM and what not to. Without any swap, it has fewer choices.

With or without swap, when evicting pages isn't sufficient, the operating system will start by refusing to permit operations that require memory (such as mmap and fork) to succeed. However, sometimes that's not enough and processes have to be killed.