Connection remains flagged as ESTABLISHED even if host is unconnected
I am working on a Linux embedded distribution and I need to monitor incoming and outcoming connections (if they are still actives etc.) and I am experiencing issues because I noticed that, even if the remote host turn off the connection, I can see the connection as ESTABLISHED through netstat.
I have two different versions of this Linux embedded system: one which uses an old 2.6.34 kernel and it works fine, while the one which is failing uses a kernel 3.18.18.
Looking for more information I suspect I had to set/change TCP timeouts, so I configured them through syctl.conf but doesn't change anything. The system which uses 2.6.34 kernel doesn't have TCP timeouts configured, so it makes me think that TCP are not the root of my problem.
Do you have an idea about what could be the reason of the fact that the connection remain as ESTABLISHED?
Solution 1:
If a host disconnects from the other and you still see its connection as ESTABLISHED, it's probably related to the fact it's not honoring the TCP protocol and not closing the connection cleanly.
The netstat output is an interpreter of the current state of TCP connections. If a client wants to disconnect/close the socket that has previously been open and established, they should notify this to the remote system. This is done sending the FIN request to the other node (more info here), in this case, the server.
It they fail to do so, the client indeed disconnects, but the remote server keeps thinking that the client is still connected and thus keeping their state as ESTABLISHED, and that's where the tcp_keepalive_time parameter join the equation. As no further packets will be received, the kernel will wait the specified time to this parameter to time-out the connection and forcibly close it.
You can debug this issue using the tcpdump tool. You could simply listen to the connection from the client host on the server side and check if it fails to send the FIN request.
tcpdump host X.X.X.X and port Y