Does linux have any measures to protect against fork bombs?

#include <unistd.h>
int main(int argc, char* argv[]) {
  while(1)
  {
    fork();
  } 
}

I run this program on my linux, nothing outputs on the terminal, the OS seems go dead. Does linux have any protection measure to such program which can run out of memory?

Solution 1:

This is known as a fork bomb.

Does linux have any protection measure to such program which can run out of memory?

Not really. Each fork produces a new process, with its own virtual address space and memory usage. So each copy is relatively small. Eventually, you'll use up all the physical+swap memory on the system, and the out-of-memory (OOM) killer will start killing individual processes. But the fork bomb will still be creating processes just as fast (if not faster).

One way to prevent this occurring in the first place is to limit the number of user processes, using ulimit -u (assuming you're using Bash; other shells will have equivalents).

Solution 2:

Yes, although it may not be enabled by default on your system. The setrlimit system call defines system limits -- including the number of processes per user.

Let's look at it first in the kernel API (since you mentioned "linux"): you can use the manpage for setrlimit, which will tell you to do something like

#include <sys/resource.h>
...

struct rlimit  r;

rnew.r_cur = 40;
rnew.r_max = 50;
setrlimit(RLIMIT_NPROC,&r);

This will set the maximum processes per user (RLIMIT_NPROC) to 40 (soft limit) and 50 (hard limit).

Now, from the shell, if you use bash, you can use the ulimit built-in command:

ulimit -u
29089

You can set the limit by passing it as an argument:

ulimit -u 100

ulimit --help will show you that there are several other limits you can set (one which may be of interest is the maximum number of file descriptors used by the user).

Solution 3:

It depends if you want to use it on user level or system level. On user level the ulimit(or corresponding commands for other shells) would be easiest solution.

However on system level there are mechanisms to prevent malicious users (or just not using ulimit) from stopping the system. Linux cgroups mechanism can limit the resources on per-group basis. You can force (by pam_systemd machanism) the user session to be in specific group. This have other benefits for, for example, CPU scheduler.