Linux TCP: high Send-Q on sender, zero Recv-Q on receiver

How can it be that:

1. There is a TCP socket between two machines
2. After some succesful bidirectional communications, sender application is stuck on writing to the socket and receiver on reading from it
3. netstat reports high Send-Q (a few megabytes) for the socket on the sender (and the value does not change even after a couple of hours of waiting)
4. netstat reports zero Recv-Q for the socket on the receiver
5. tcpdump reports that the only activity on the socket is a periodic (biminutely) ACK with no data from the sender and immediate ACK response with no data from the receiver

Why doesn't the sender machine attempt to send queued data to the receiver?

Solution 1:

I my case, client was writing data in chunks of 8KB and server was trying to read 8KB and server would then write it to RAID0 disks. For uploading large files, I faced a similar situation and increasing the amount of data I was reading from socket on server side helped. I bumped up the size of internal buffer that was reading from socket to 1MB (from 8 kB) and it helped. I don't know for sure whether it was because of RAID or tcp but it could be another thing you might want to try out.

Solution 2:

This is more likely caused by other problem, but below might help if you haven't tried (these numbers are examples, find your own numbers):

1. Estimate your sender and receiver file system read/write speed as well as network speed, and set appropriate bandwidth limit in rsync: --bwlimit=1024 (1024 KBps)
2. If sender and receiver have dedicated NIC in this local network, do yourself a favor, increase MTU on these NICs: ifconfig eth1 mtu 65744
3. Increase sender transmission queue length: ifconfig eth1 txqueuelen 4096
4. Increase kernel send/receive memory:
   add these to /etc/sysctl.conf file
   net.core.wmem_max=16777216
   net.core.rmem_max=16777216
   net.ipv4.tcp_rmem=4096 262144 16777216
   net.ipv4.tcp_wmem=4096 262144 16777216
   run sysctl -p afterwards.
5. If you rsync a very large file system, make sure fs.file-max is large enough,
   to check it: sysctl  fs.file-max
   to increase it, add a line fs.file-max=327679 to the file /etc/sysctl.conf,
   on your rsync user, run: ulimit -n 327679