Number of file descriptors: different between /proc/sys/fs/file-nr and /proc/$pid/fd?
Solution 1:
lsof
only lists the Process ID. To get info about threads, you should use ps -eLf
. According to the man proc
:
/proc/[pid]/task (since Linux 2.6.0-test6) This is a directory that contains one subdirectory for each thread in the process. The name of each subdirectory is the numerical thread ID ([tid]) of the thread (see gettid(2)). Within each of these subdirectories, there is a set of files with the same names and contents as under the /proc/[pid] directories. For attributes that are shared by all threads, the contents for each of the files under the task/[tid] subdirectories will be the same as in the corresponding file in the parent /proc/[pid] directory (e.g., in a multithreaded process, all of the task/[tid]/cwd files will have the same value as the /proc/[pid]/cwd file in the parent directory, since all of the threads in a process share a working directory). For attributes that are distinct for each thread, the corresponding files under task/[tid] may have different values (e.g., various fields in each of the task/[tid]/status files may be different for each thread).
In a multithreaded process, the contents of the /proc/[pid]/task directory are not available if the main thread has already terminated (typically by calling pthread_exit(3)).
I would calculate the number of open file descriptors by running:
ps -eL | awk 'NR > 1 { print $1, $2 }' | \
while read x; do \
find /proc/${x% *}/task/${x#* }/fd/ -type l; \
done | wc -l
The result is 17270.
Let's see how many file descriptors allocated since boot:
cat /proc/sys/fs/file-nr
11616 0 398855
Why there is an excess of number of file descriptors in /proc/[pid]/task/[tid]/fd
over the number of allocated file handles in /proc/sys/fs/file-nr
? I suppose that they are created by fork
ed child processes:
man fork
:
The child inherits copies of the parent's set of open file descriptors.
man pthreads
:
POSIX.1 also requires that threads share a range of other attributes (i.e., these attributes are process-wide rather than per-thread): - process ID
parent process ID
process group ID and session ID
controlling terminal
user and group IDs
open file descriptors
Solution 2:
http://www.netadmintools.com/part295.html Some of the open files which are not using file descriptors: library files, the program itself (executable text), and so on as listed above. These files are accounted for elsewhere in the kernel data structures (cat /proc/PID/maps to see the libraries, for instance), but they are not using file descriptors and therefore do not exhaust the kernel's file descriptor maximum.