How to estimate how long an rsync of 427GB directory comprises of 470k subdirectories and 4476k files would take?

Solution 1:

Time for some rough Fermi estimates with GNU units as a calculator.

Pretend this was one big archive and it could copy sequentially. 1 Gb link, say with overhead the transfer gets 70% of that.

You have: (.7 / 8) GB /s
You want: hr / 427 GB
        reciprocal conversion
        * 1.3555556
        / 0.73770492

But it took way longer than an hour and a half. With lots of small files, metadata IO dominates transfer times, and these are small on average:

You have: 427 GB / 4476474
You want: kB
        * 95.387575
        / 0.010483546

Assume rsync has to do an IO per file or directory to check metadata, and another IO to copy it. You didn't describe storage, but say worst case of a SATA spinner, so 50 IOPS.

You have: 50 / s
You want: hr / (2 * 5000000)
        reciprocal conversion
        * 55.555556
        / 0.018

Two and a quarter days is actually low, if you said you stopped it on the 3rd day.

My educated guesses will get a lot of things wrong: system load, storage capability, how rsync's algorithms perform with this many files, how many files were deleted in the target. But it helps to get close to the order of magnitude of work done, then compare to the system's limits.

About copying this faster: copy archives or images, not files. Much less file metadata, and a sequential copy. For example, take a LVM snapshot, and use dd to pipe the entire file system over the network. Not nearly as flexible as rsync, can only copy over the entire volume.

Solution 2:

Include the option -v and --progress

This will display the count-down timer while the copy process is in progress.

user@ubuntu:/opt/$ rsync -v --progress test.iso /tmp/
test.iso
  1,459,486,720 100%   51.41MB/s    0:00:27 (xfr#1, to-chk=0/1)

sent 1,459,843,134 bytes  received 35 bytes  53,085,206.15 bytes/sec
total size is 1,459,486,720  speedup is 1.00

In this example, the copy process will take 27 seconds (0:00:27) to complete. The count-down timer started from 27 seconds (copy progress 0%) and went down all the way to zero seconds (copy progress 100%). After completion, the 27 seconds was re-displayed as an indicator that the whole process took 27 seconds in total.

Another option is to pipe the rsync command to pv command

Reference: Here