Why does't rsync use delta-transfer for local files?

Solution 1:

According to the manpage, psusi is right:

-W, --whole-file: The transfer may be faster if this option is used when the bandwidth between the source and destination machines is higher than the bandwidth to disk (especially when the "disk" is actually a networked filesystem). This is the default when both the source and destination are specified as local paths, but only if no batch-writing option is in effect.

Solution 2:

The straight-forward answer to this question is:

Use the --no-W flag to force delta compression, no matter local or remote.

Update: It looks like there is more to the story. The delta compression seems to be enabled only between receive and transmit process of rsync. When outputting the file to the file-system, rsync may still write out the whole file(s), even with delta compression on.

See "Wakan Tanka's" investigation here.

Update 2: The --inplace option writes only the changed parts of the file. However note that it conflicts with --sparse and is not recommended, by the manual, when the destination file is being used.

Solution 3:

By default, rsync first creates a new copy of the target file and then replaces it, for various safety reasons. You can override this by specifying --inplace along with --no-whole-file. This tells rsync to do an in-place-edit of the target file, accepting the various risks (typically minor for this situation) as documented in the man page.

Solution 4:

By default rsync creates a copy of the file at the destination and then atomically replaces the original with the new copy. This is done for safety reasons. What you're looking for is the --inplace option, which will cause rsync to modify only the portions of the destination file which have changed relative to the source.

For the O.P's use-case, I recommend turning off pre-allocation as well, so that a sparse copy can be synced, which will be much faster. For downloads, don't worry about fragmentation unless you're using a very ancient filesystem like VFAT. Media files in particular are not read at the maximum performance of the storage media, so defragmenting them is a wasted effort.

To copy your downloads directory sparsely to the destination volume, I recommend these flags and operations, in this order:

rsync --ignore-existing -vxaHAXS /source /destination
rsync --inplace -vxaHAX /source /destination

The first pass will copy new files sparsely to the destination The second pass will update existing files in-place, copying only the changes

Since it's doing sparse and in-place delta copies, you can run this repeatedly without incurring much extra IO. Even if you have 20 torrents running simultaneously it's won't amplify the writes at the destination, or thrash the source/dest volumes.