Too many open files - how to find the culprit

You can use lsof to understand who's opening so many files. Usually it's a (web)server that opens so many files, but lsof will surely help you identify the cause.

Once you understand who's the bad guy you can

• kill the process/stop the program
• raise the ulimit

If output from lsof is quite huge try redirecting it to a file and then open the file

Example (you might have to Ctrl+C the first command)

lsof > ~/Desktop/lsof.log
cat ~/Desktop/lsof.log | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20
vim ~/Desktop/lsof.log

In case anyone else needs it...

ulimit -a

Will display all current limits. Specifically ulimit -n 70000 will set the file descriptor limit.

Also...

cat /proc/sys/fs/file-max

Will display/set the kernel limit if edited.

sudo echo 200000 > /proc/sys/fs/file-max

A much more detailed explanation can be found at...

How do I increase the open files limit for a non-root user?

While ulimit can be used to determine how many files are allowed to be open per process you may want to find the culprit.

@itsadok @Tyler Collier @gaoithe in comments to other answers highlight that sorting and counting which process has the most files open is the best thing to do here:

sudo lsof | head -1 | awk '{ print "COUNT " $2 " " $1; }' && sudo lsof +c 0 | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20

Above command:

• Gives output a header
• lsof gives list of open files
• +c 0 option instructs lsof to print full command
• awk '{ print $2 " " $1; }' prints PID and COMMAND column in results
• sort -rn sorts results so identical entries are next to each other (needed for uniq -c to work properly)
• uniq -c counts files open by PID/command
• sort -rn sorts results by count
• head -20 shows top 20 files open by PID/command

Note: This will count includes "files" that don't count towards limits

You may want to investigate even further by looking at limits for a PID, # of files open for a specific PID, and limit lsof to only count files that count towards limit - see https://serverfault.com/a/964752/152562.