samba stuck at maximum of 1024 open files

Solution 1:

Ok I have solved my issue, and in doing so come to a better understanding of how the ulimits work, at least in Ubuntu. There were a number of issues and I think I have sorted them all out.

First problem, and a silly one: nofiles should be nofile in /etc/security/limits.conf

Another more significant oversight: While I had ensured pam_limits.so was included in /etc/pam.d/common-session, I didn't notice that there was also /etc/pam.d/common-session-noninteractive. The latter file was the one that samba was using.

Fixing that issue appears to have fixed samba, which can now open as many file descriptors as it likes. Windows copies complete successfully. Also note: Samba does indeed use the appropriate user's ulimit, not the ulimits the smbd process started with, nor root's ulimit. /etc/security/limits.conf is the place to set this, once you have properly configured either (both?) /etc/pam.d/common-session-noninteractive and /etc/pam.d/samba to use pam_limits.so

As for the other issue, where my user was stuck at 1024 hard/1024 soft limits, that was a combination of a few issues. First and foremost, despite having /etc/pam.d/sshd the ssh daemon does not use PAM unless you modify /etc/ssh/sshd_config to have "UsePAM yes". The default is "no", and without using PAM, pam_limits.so (which is responsible for applying limits.conf) does not even come into play.

Instead, the default ulimits for non-PAM logins seem to inherit from pid 1 (typically "init"). You can check those default pid 1 limits with cat /proc/1/limits. Unfortunately, as far as I can tell, those limits are set as defaults in the kernel. There does not seem to be any way to modify them short of recompiling the kernel, or convincing the non-PAM application to use PAM.

I also just want to offer the advice that cat /proc/<anypid>/limits is a great way to debug the limits of any specific process you might be having trouble with. I wish I had discovered that sooner.

Solution 2:

I was working with ubuntu 14.04 lts and needed several hours to realize the following:

• samba-ad-dc is startet with upstart, so settings in /etc/security/limits.conf are irrelevant.
• limits are defined with the stanza in the file /etc/init/samba-ad-dc.conf

• there had to be a line inserted with the following code:

  limit nofile 16384 16384

After a restart all works fine, ps ax tells you the process id of samba and cat /proc/799/limits shows the correct Max open file limits of the process 799 (exchange 799 with your process id).