netstat shows a listening port with no pid but lsof does not

Solution 1:

From data you provided I'd say it's related to some NFS mounts or something using RPC.

you can check with rpcinfo -p for ports that might be used by some of RPC related services.

Here is how it looks on my system

# netstat -nlp | awk '{if ($NF == "-")print $0}'
tcp        0      0 0.0.0.0:55349           0.0.0.0:*               LISTEN      -               
udp        0      0 0.0.0.0:18049           0.0.0.0:*                           - 

# rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  10249  status
    100024    1   tcp  10249  status
    100021    1   udp  18049  nlockmgr
    100021    3   udp  18049  nlockmgr
    100021    4   udp  18049  nlockmgr
    100021    1   tcp  55349  nlockmgr
    100021    3   tcp  55349  nlockmgr
    100021    4   tcp  55349  nlockmgr

Solution 2:

Some processes/pids are only available to root. Try

sudo netstat -antlp

it should return the pid of every open port that's not in a TIME_WAIT state

or, if you want to know process ID related to specific port (let us say 8765 for example) use the code

netstat -tulpn | grep :8765

Solution 3:

Based on hint from @user202173 and others I have been able to use the following to track down the process that owns a port even when it is listed as - in netstat.

Here was my starting situation. sudo netstat shows port with PID/Program of -. lsof -i shows nothing.

$ sudo netstat -ltpna | awk 'NR==2 || /:8785/'
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 :::8785                 :::*                    LISTEN      -
tcp6       1      0 ::1:8785                ::1:45518               CLOSE_WAIT  -
$ sudo lsof -i :8785
$

Now let's go fishing. First let's get the inode by adding -e to our netstat call.

$ sudo netstat -ltpnae | awk 'NR==2 || /:8785/'
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp6       0      0 :::8785                 :::*                    LISTEN      199179     212698803   -
tcp6       1      0 ::1:8785                ::1:45518               CLOSE_WAIT  0          0           -

Next use lsof to get the process attached to that inode.

$ sudo lsof | awk 'NR==1 || /212698803/'
COMMAND      PID    TID                USER   FD      TYPE             DEVICE   SIZE/OFF       NODE NAME
envelope_ 145661 145766               drees   15u     IPv6          212698803        0t0        TCP *:8785 (LISTEN)

Now we know the process id so we can look at the process. And unfortunately it's a defunct process. And its PPID is 1 so we can't kill its parent either (see How can I kill a process whose parent is init?). In theory init might eventually clean it up, but I got tired of waiting and rebooted.

$ ps -lf -p 145661
F S UID         PID   PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 Z drees    145661      1  2  80   0 -     0 exit   May01 ?        00:40:10 [envelope] <defunct>

Solution 4:

I don't know what these are specifically, but kernel modules (NFS for example) do not have a PID to associate with these sockets. Look for something suspect in lsmod.