Showing total progress in rsync: is it possible?
There is now an official way to do this in rsync (version 3.1.0 protocol version 31, tested with Ubuntu Trusty 14.04).
#> ./rsync -a --info=progress2 /usr .
305,002,533 80% 65.69MB/s 0:00:01 xfr#1653, ir-chk=1593/3594)
I tried with my /usr folder because I wanted this feature for transferring whole filesystems, and /usr seemed to be a good representative sample.
The --info=progress2 gives a nice overall percentage, even if it's just a partial value. In fact, my /usr folder is more than 6 gigs:
#> du -sh /usr
6,6G /usr/
and rsync took a lot of time to scan it all. So almost all the time the percentage I've seen was about 90% completed, but nonetheless it's comforting to see that something is being copied :)
References:
- https://stackoverflow.com/a/7272339/1396334
- https://download.samba.org/pub/rsync/NEWS#3.1.0
The following applies to rsync version 3.0.0 and above. The options described below were introduced in that release on March 1st, 2008.
Along with --info=progress2 you can also use --no-inc-recursive option (or its shorter --no-i-r alias) to disable incremental recursion.
This will build the entire file list at the beginning, rather than incrementally discovering more files as the transfer goes on. Since it will know all files before starting, it will give a better report of the overall progress. This applies to the number of files - it does not report any progress based on file sizes.
This involves a trade-off. Building the entire file list ahead of time is more memory-costly and it can significantly delay the start of the actual transfer. As you would expect, the more files there are, the longer the delay will be and the more memory it will require.
The following is from the rsync manual (source - http://rsync.samba.org/ftp/rsync/rsync.html ):
-r, --recursive
This tells rsync to copy directories recursively. See also --dirs (-d). Beginning with rsync 3.0.0, the recursive algorithm used is now an incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed. This incremental scan only affects our recursion algorithm, and does not change a non-recursive transfer. It is also only possible when both ends of the transfer are at least version 3.0.0.
Some options require rsync to know the full file list, so these options disable the incremental recursion mode. These include: --delete-before, --delete-after, --prune-empty-dirs, and --delay-updates. Because of this, the default delete mode when you specify --delete is now --delete-during when both ends of the connection are at least 3.0.0 (use --del or --delete-during to request this improved deletion mode explicitly). See also the --delete-delay option that is a better choice than using --delete-after.
Incremental recursion can be disabled using the --no-inc-recursive option or its shorter --no-i-r alias.
See also https://rsync.samba.org for specific version differences (scroll down and check out the Release News links).