Why is rpc.lockd obscured from netstat/lsof output?
Prologue:
On a number of machines, which happen to act as NFS clients, netstat
reports two ports that are open with no PID listed for an associated daemon. Ordinarily this might be a bit concerning.
# netstat -lnp | egrep -- '- +$'
tcp 0 0 0.0.0.0:57448 0.0.0.0:* LISTEN -
udp 0 0 0.0.0.0:48933 0.0.0.0:* -
Additionally netcat
confirms that the TCP port really is open.
# nc -v localhost 57448
localhost [127.0.0.1] 57448 (?) open
^C
Yet lsof
reports nothing for these two ports. Intrigue grows.
# lsof -i TCP:57448 -i UDP:48933
However rpcinfo
finally points us in the right direction. It's being held open by nlockmgr
, aka lockd
for NFS. Call off the search.
# rpcinfo -p | egrep '57448|48933'
100021 1 udp 48933 nlockmgr
100021 3 udp 48933 nlockmgr
100021 4 udp 48933 nlockmgr
100021 1 tcp 57448 nlockmgr
100021 3 tcp 57448 nlockmgr
100021 4 tcp 57448 nlockmgr
It becomes clear that lockd
/rpc.lockd
is called when mounting an NFS export. This is a kernel thread (is it always?) which binds itself to one TCP and one UDP port in the ephemeral range. The ports are typically reconfigurable with the fs.nfs.nlm_tcpport
and fs.nfs.nlm_udpport
sysctls.
Questions:
I am intrigued though. Would love some kernel-internals insight as to..
Why isn't the kernel thread's PID visible from
netstat
?Why aren't the bound ports visible from
lsof
?
netstat and lsof both crawl /proc/<pid>/fd
to associate processes to ports/sockets, and /proc/<pid>/fd
doesn't get populated for kernel threads AFAIK.
lockd is always a kernel thread - at least on modern (newer than say 2.2) kernels.