Should I use rsync compression over a gigabit LAN?

Solution 1:

During an rsync transfer are you CPU bound, or is your link saturated.

• If your link is saturated, but your CPU is idle, then compress.
• If your CPU is maxed, and your link isn't, then do not compress.

Solution 2:

I work this quite often, migrating data between servers and during impossibly-long data transfers...

The short answer is to test with your specific data... This is really easy to do. Try a LAN transfer with compression off, then try it with compression on...

In my experience with production data sets across a few environments, on a GigE or greater connection, enabling rsync compression throttled transfers to ~40 Megabytes/second (or 33% of the theoretical maximum speed of a Gigabit link).

That rate varied with the size and number of files being transferred as well (lots of small files, for instance, lowered overall rsync throughput). The storage subsystems of the sender and receivers also mattered.

Recently, I've been using a modified rsync for LAN transfers. By using the UDR library with rsync (allow UDP transfer), and disabling compression and encryption, I can get wire-speed rsync transfers with mixed file types, regardless of the number of files.

Solution 3:

Like Zoredache said, if your link isnt saturated no -z. Also, something else you can consider to help manage long transfers is the bwlimit flag.