Practical maximum open file descriptors (ulimit -n) for a high volume system

We recently began load testing our application and noticed that it ran out of file descriptors after about 24 hours.

We are running RHEL 5 on a Dell 1955:

CPU: 2 x Dual Core 2.66GHz 4MB 5150 / 1333FSB RAM: 8GB RAM HDD: 2 x 160GB 2.5" SATA Hard Drives

I checked the file descriptor limit and it was set at 1024. Considering that our application could potentially have about 1000 incoming connections as well as a 1000 outgoing connections, this seems quite low. Not to mention any actual files that need to be opened.

My first thought was to just increase the ulimit -n parameter by a few orders of magnitude and then re-run the test but I wanted to know any potential ramifications of setting this variable too high.

Are there any best practices towards setting this other than figuring out how many file descriptors our software can theoretically open?

These limits came from a time where multiple "normal" users (not apps) would share the server, and we needed ways to protect them from using too many resources.

They are very low for high performance servers and we generally set them to a very high number. (24k or so) If you need higher numbers, you also need to change the sysctl file-max option (generally limited to 40k on ubuntu and 70k on rhel) .

Setting ulimit:

# ulimit -n 99999

Sysctl max files:

#sysctl -w fs.file-max=100000

Also, and very important, you may need to check if your application has a memory/file descriptor leak. Use lsof to see all it has open to see if they are valid or not. Don't try to change your system to work around applications bugs.

You could always just

cat /proc/sys/fs/file-nr

During the 'high load' situation to see how many file descriptors are in use.

As to a maximum - it just depends on what you are doing.

If the file descriptors are tcp sockets, etc, then you risk using up a large amount of memory for the socket buffers and other kernel objects; this memory is not going to be swappable.

But otherwise, no, in principle there should be no problem. Consult the kernel documentation to try to work out how much kernel memory it will use, and/or test it.

We run database servers with ~ 10k file descriptors open (mostly on real disc files) without a major problem, but they are 64-bit and have loads of ram.

The ulimit setting is per-process, but there is a system-wide limit as well (32k I think by default)