How to stop and detect the fork bomb

Solution 1:

One way is to limit the number of processes , a user can run.

Just login as root , and edit this file , to add users and configure , their limit.

# vi /etc/security/limits.conf

Add this line to the file

john hard nproc 10

Now user john can create only 10 processes.

Solution 2:

To stop a running fork bomb you might be able to use killall <name> to kill all processes of the bomb. However, since a fork bomb usually results in an incredibly high load on the system you might not be able to SSH into it or execute that. So a reboot might be necessary or at least much faster.

If every user has his own account on the system you can simply check everyone's home directory and search for the executable. Chances are good he also uploaded the source code so finding it shouldn't be too hard. If it was a shared account for all students you are out of luck. Especially after the telnet or ssh session of the user terminated you have no chance to find out who started it.

However, instead of punishing the user who detonated that fork bomb you should rather fix the system's configuration to disarm fork bombs. You can set per-user process limits using /etc/security/limits.conf and thus prevent a fork bomb from getting out of control - with e.g. just 50 processes a fork bomb won't do much damage.