rsync does not sync files

Solution 1:

Finally, the problem has been resolved. It's hard to believe but it was incorrect MTU set on the main network interface. After changing MTU to 1460 the sync process has been started and completed immediately. Thanks all for answers.

Solution 2:

I ran into the same symptoms recently. Both sides strace showed they were in select() waiting for the other side. I then noticed that the server end had a large send queue in netstat, so I began looking for network-level solutions. I tried reducing the MTU as above, but it didn't make a difference. Then I disabled SACK on both sides, and rsync began to work again:

echo 0 > /proc/sys/net/ipv4/tcp_sack

There's some discussion out there of Cisco bugs with selective ack and sequence number randomization, which is at least a plausible reason for this making a difference.