parallel with copy command is very slow to clone 2 directories

I am on MacOS Catalina. This command is very slow :

parallel -j32 cp -rpf {} /Users/henry/ ::: /Volumes/henry/

The goal is to clone the /Volume/henry to /Users/henry directory.

/Volume/henry is shared across my network.

Could anyone tell me what's wrong in my command : I have 1.3 TB to clone.

Any help is welcome.

Solution 1:

The command should likely be adjusted like this:

parallel -j32 cp -rpf {} /Users/henry/ ::: /Volumes/henry/*

Note the asterisk, this will allow parallel to parallelize across the folders/files present to be copied within the top level of /Volumes/henry/.

With the above command, there will be as many concurrent copies as /Volumes/henry/* expands to, which is why the original command (without the asterisk) was slow (it was performing one parallel copy).

However, since you are copying so much information, I'd recommend an alternate approach, and use rsync rather than cp. rsync might be slightly slower, but it verifies that copies complete properly, so would help guarantee that the destination matches the source, and it is resumable in case the process prematurely ends.

Something like this should work:

parallel -j32 rsync -av {} /Users/henry/ ::: /Volumes/henry/*

Finally, you may need to play around with the number of maximum threads, 32 might be too high depending on the I/O of your disks on the source and destination, and depending on how many folders/files there are in the folder /Volumes/henry/. Since this copy is network bound, 32 threads may not speed the copy up as much as is hoped.