How to find other end of unix socket connection?
I have a process (dbus-daemon) which has many open connection over UNIX sockets. One of these connections is fd #36:
=$ ps uw -p 23284
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
depesz 23284 0.0 0.0 24680 1772 ? Ss 15:25 0:00 /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
=$ ls -l /proc/23284/fd/36
lrwx------ 1 depesz depesz 64 2011-03-28 15:32 /proc/23284/fd/36 -> socket:[1013410]
=$ netstat -nxp | grep 1013410
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
unix 3 [ ] STREAM CONNECTED 1013410 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
=$ netstat -nxp | grep dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1013953 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1013825 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1013726 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1013471 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1013410 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1012325 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1012302 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1012289 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1012151 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011957 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011937 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011900 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011775 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011771 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011769 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011766 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011663 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011635 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011627 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011540 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011480 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011349 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011312 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011284 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011250 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011231 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011155 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011061 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011049 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011035 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1011013 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1010961 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
unix 3 [ ] STREAM CONNECTED 1010945 23284/dbus-daemon @/tmp/dbus-3XDU4PYEzD
Based on number connections, I assume that dbus-daemon is actually server. Which is OK. But how can I find which process is connected to it - using the connection that is 36th file handle in dbus-launcher? Tried lsof and even greps on /proc/net/unix but I can't figure out a way to find the client process.
Solution 1:
Quite recently I stumbled upon a similar problem. I was shocked to find out that there are cases when this might not be possible. I dug up a comment from the creator of lsof (Vic Abell) where he pointed out that this depends heavily on unix socket implementation. Sometimes so called "endpoint" information for socket is available and sometimes not. Unfortunatelly it is impossible in Linux as he points out.
On Linux, for example, where lsof must use /proc/net/unix, all UNIX domain sockets have a bound path, but no endpoint information. Often there is no bound path. That often makes it impossible to determine the other endpoint, but it is a result of the Linux /proc file system implementation.
If you look at /proc/net/unix you can see for yourself, that (at least on my system) he is absolutelly right. I'm still shocked, because I find such feature essential while tracking server problems.
Solution 2:
This answer is for Linux only.
Update for Linux 3.3: As Zulakis wrote in a separate answer (+1 that), you can use ss from iproute2 to get a pair of inode numbers for each socket connection identifying local end and peer. This appears to be based on the same machinery as sock_diag(7) with the UNIX_DIAG_PEER attribute identifying the peer. An answer by Totor over at Unix & Linux Stack Exchange links to the relevant commits in kernel and iproute2 and also mentions the need for the UNIX_DIAG kernel config setting.
Original answer for Linux pre 3.3 follows.
Based on an answer from the Unix & Linux Stack Exchange, I successfully identified the other end of a unix domain socket using in-kernel data structures, accessed using gdb and /proc/kcore. You need to enable the CONFIG_DEBUG_INFO and CONFIG_PROC_KCORE kernel options.
You can use lsof to get the kernel address of the socket, which takes the form of a pointer, e.g. 0xffff8803e256d9c0. That number is actually the address of the relevant in-kernel memory structure or type struct unix_sock. That structure has a field called peer which points at the other end of the socket. So the commands
# gdb /usr/src/linux/vmlinux /proc/kcore
(gdb) p ((struct unix_sock*)0xffff8803e256d9c0)->peer
will print the address of the other end of the connection. You can grep the output of lsof -U for that number to identify the process and file descriptor number of that other end.
Some distributions seem to provide kernel debug symbols as a separate package, which would take the place of the vmlinux file in the above command.