Odd TCP termination sequence
While troubleshooting another thing, I've noticed an odd pattern of TCP closes.
Packets: http://www.cloudshark.org/captures/7590ec4e6bef
What's happening is that the last few packets of the close sequence are getting retransmitted for some odd reason. The pattern is in the cloudshark link, but for posterity here is a summary:
- Source -> Syn
- Dest -> Ack
- Source -> SynAck
- Data
- Data
- Source -> Fin/Ack
- Dest -> Psh/Ack (6)
- Dest -> Fin/Ack
- Source -> Ack (7)
- Source -> Ack (8)
- [At this point the connection should be closed on both sides. But it isn't.]
- [+200ms] Dest -> Fin/Ack
- Source -> Ack (8 & 12)
Something on the Destination system is waiting 200ms before reissuing the Fin/Ack packet. This is pretty consistent across multiple transactions. More damning, this pattern replicates on both sides of the transaction: it shows up in captures on both hosts. It isn't as simple as the Fin/Ack packet getting dropped somewhere and getting retransmitted. Or maybe it is getting dropped, but on a level above where tcpdump operates.
The 200ms delay makes me think TCP Delayed Ack is involved here, but I'm having trouble figuring out why it would.
Is above tcpdump even a thing?
Is this a normal connection pattern for RHEL6 systems?
Solution 1:
Note that the PSH/ACK packet in frame #2 of your capture is carrying 37 bytes of data. It is not a response to the FIN request from 10.232.16.133. The response follows in frame #3, the ACK to that is in frame #5.
What seems to be happening now is that this last ACK does not make it to the destination, so the FIN/ACK sent in frame #3 is retransmitted in frame #6. The ~200 ms delay you are seeing here is not the delayed ack but more likely the retransmission timeout. You should be able to verify this by looking at the packet capture from the other side of the connection where you never should be seeing the last ACK arriving.
If you see this behavior consistently on all connections (and possibly in both directions), an explanation crossing my mind is that some state-keeping packet filter in the path is swallowing the last ACK.