How to interpret the output of netstat -o / netstat --timers
netstat -o includes some timer information in the output but I haven't found an explanation of the output in the Timer column anywhere.
Can anybody explain this or point to an explanation?
This ist what the ouptput of netstat -o looks like (on Ubuntu 8.04).
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State Timer
tcp 0 0 192.168.22.1:443 111.111.11.210:5804 ESTABLISHEDkeepalive (6176.47/0/0)
tcp 0 0 192.168.22.1:443 192.168.22.253:48379 TIME_WAIT timewait (36.57/0/0)
tcp 0 924 192.168.22.1:47763 10.9.169.60:443 ESTABLISHEDon (0.34/0/0)
tcp 0 0 192.168.22.1:443 192.168.111.99:4059 ESTABLISHEDkeepalive (6963.60/0/0)
tcp 0 0 192.168.22.1:443 192.168.111.74:1729 ESTABLISHEDkeepalive (1393.60/0/0)
tcp 0 0 192.168.56.1:42204 10.9.169.60:443 ESTABLISHEDoff (0.00/0/0)
tcp 0 0 192.168.56.1:42207 10.9.169.60:443 ESTABLISHEDoff (0.00/0/0)
tcp 0 940 192.168.22.1:42186 10.9.169.60:443 ESTABLISHEDon (0.28/0/0)
tcp 0 0 192.168.22.1:443 192.168.22.253:48367 TIME_WAIT timewait (31.57/0/0)
tcp 0 0 192.168.22.1:42234 10.9.169.60:443 ESTABLISHEDoff (0.00/0/0)
tcp 0 0 192.168.22.1:42209 10.9.169.60:443 ESTABLISHEDoff (0.00/0/0)
Solution 1:
The first number is clearly the countdown timer. Depending on status type, once the timer times out, it will retry and send out another FIN or whatever needed packet to try again. So the second number keeps track of the number of retries. Notice that the timer increases because TCP has backoff timers in case there is a traffic condition. The backoff timer avoids excessively frequent retries. Each failure results in a larger and larger backoff. The backoff could be exponential growth or linear, depending on TCP stack. I have not figured out what the 3 figure is for. Always zero in my displays.
Solution 2:
The timer column has two fields (from your o/p above):
keepalive (6176.47/0/0)
<1st field> <2nd field>
The 1st field can have values:keepalive - when the keepalive timer is ON for the socketon - when the retransmission timer is ON for the socketoff - none of the above is ON
The 2nd field has THREE subfields:
(6176.47/0/0) -> (a/b/c)
a=timer value (a=keepalive timer, when 1st field="keepalive"; a=retransmission timer, when 1st field="on")b=number of retransmissions that have occurredc=number of keepalive probes that have been sent
For example, I had two sockets opened between a client & a server (not loopback). The keepalive setting are:
KEEPALIVE_IDLETIME 30
KEEPALIVE_NUMPROBES 4
KEEPALIVE_INTVL 10
And I did a shutdown of the client machine, so at the server side I ran a netstat & the output was:
Port1:
netstat -c --timer | grep "192.0.0.1:43245 192.0.68.1:49742"
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (1.92/0/0)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (0.71/0/0)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (9.46/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (8.30/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (7.14/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (5.98/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (4.82/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (3.66/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (2.50/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (1.33/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (0.17/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (9.01/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (7.75/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (6.47/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (5.29/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (4.08/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (2.89/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (1.73/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (0.54/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (9.38/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (8.23/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (7.08/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (5.93/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (4.76/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (3.62/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (2.48/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (1.32/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (0.13/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (8.98/0/4)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (7.78/0/4)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (6.62/0/4)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (5.45/0/4)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (4.29/0/4)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (3.14/0/4)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (1.99/0/4)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (0.85/0/4)
You can see above that the server sent FOUR keepalive probes, each after 10 sec and since it is not getting any response, the number-of-probes-sent count increases, and after 4, it disconnects from the client.
Port 2:
For the 2nd connection, the socket was identical, except for, my application at server side was trying to send some message after the client went down & before the keepalive expired:
netstat -c --timer | grep "192.0.0.1:36483 192.0.68.1:43881"
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (8.18/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (7.00/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (5.86/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (4.71/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (3.55/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (2.40/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (1.21/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (0.05/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (8.91/0/2)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (7.75/0/2)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (6.56/0/2)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (5.39/0/2)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (4.14/0/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.21/2/2) // <---- retransmission timer kicks in
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.68/3/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.74/4/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.59/4/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.43/4/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.28/5/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.11/5/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.95/6/2)
. . . . .
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.65/249/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.58/250/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.48/250/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.36/250/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.26/251/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.15/251/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (3.01/252/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.92/252/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.84/252/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.72/253/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.64/253/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.55/253/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.47/254/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.39/254/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.31/254/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.19/255/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.12/255/2)
As you can see, in this case things are a little different. When the client went down, my server started sending keepalive messages, but while it was still sending those keepalives, my server tried to send a message to the client. Since the client had gone down, the server couldn't get any ACK from the client, so the TCP retransmission started and the server tried to send the data again, each time incrementing the retransmit count (2nd field) when the retransmission timer (1st field) expired.
Hope this explains the netstat --timer option well.
Solution 3:
First field: timewait/keepalive
Time in seconds from when the last data was transferred until when the next TCP keepalive probe will be sent.
By default this starts at 7200s, and resets again every time more data is sent. If the value is low , for eg. 4000 seconds , it means some of the keep alive connections are hanging or doing nothing for a long period.
Connections to internal proxy or other internal processes might hang longer but this should not happen to web based connection.