Prevent fork bomb inside docker container

I am currently struggling with limiting number of processes for user which I called sandbox.

I configured processes limit inside /etc/security/limits.conf like this:

sandbox            hard    nproc            100

But if I want to connect to container as sandbox user, ssh returns:

shell request failed on channel 0

So I logged in as root and checked how many processes are running by sandbox user, but it's less than 5.

So what could be preventing me to login through ssh?
Without setting limits ssh login for sandbox user works ok.

Or is there any other way to prevent fork bomb attack?

any error when connecting to an sshd is logged into /var/log/auth.log (in debian-based OS, or security in redhat-based systems)

If it's not, set LogLevel VERBOSE in /etc/ssh/sshd_config and reload sshd. This will show you why sshd is refusing your connection next.

That said, back to your fork-bomb limiter: docker machines are based on LXC, a container system for Linux. LXC are using CGROUPS to manage resource limits for each container.

in /sys/fs/cgroups, you can setup any limit for a running LXC, and in /Var/lib/lxc/vmname/config, you can setup the limits that will be enforced at boottime.

Limiting the number of processes in a cgroup is done by the task counter subsystem (added in the Linux kernel in 2011 http://lkml.iu.edu//hypermail/linux/kernel/1109.0/01455.html )

With a recent enough linux kernel, limiting the number of allowed processes for a cgroup is therefore done by adding this kind of line into your lxc's config file :

lxc.cgroup.tasks.limit = 1024 

for a maximum of 1024 processes

(disclaimer: information not checked on a real machine though, to be confirmed)