Why is there a write speed difference between dd, cp, rsync and macOS Finder to a SMB3 drive?

Solution 1:

1. similar question but has interesting answers, may be you can check this thread especially on comment 5: https://bugzilla.samba.org/show_bug.cgi?id=8512#c5

Here quote "Peter van Hooft". Althought i'm not sure testing base on what/which linux dist. and version of rsync too. However: 1. he give us a thought to try --sparse flag if could possible to increases performance. 2. he tested on NFS protocol but met same speed issue, therefore, IT maybe not the protocol (SMB2/3, AFP, etc.) reason.

We use rsync to copy data from one file server to another using NFS3 mounts over a 10Gb link. We found that upping the buffer sizes (as a quick test) increases performance. When using --sparse this increases performance with a factor of fifty, from 2MBps to 100MBps.

2. Here is another a interesting inspection about rsync performance. https://lwn.net/Articles/400489/

LWN.net has a Conclusions that performance issue may relevant to kernel even the article posted on 2010 and we can't changed on NAS or MacOS. However, this article give us a thought that kernel issue may cause by checksum (my guessing) calculation.

One thing is clear: I should upgrade the kernel on my Mythtv system. In general, the 2.6.34 and 2.6.35-rc3 kernels give better performance than the old 2.6.27 kernel. But, tinkering or not, rsync can still not beat a simple cp that copies at over 100MiB/s. Indeed, rsync really needs a lot of CPU power for simple local copies. At the highest frequency, cp only needed 0.34+20.95 seconds CPU time, compared with rsync's 70+55 seconds.

also quote comments has this:

Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred

update 1: my mistake, this description is for --checksum. check it here: [Improved the description of the --checksum option.] PS, i don't have enough reputation to post more than 2 links.

but i can't find same description from rsync man page, so i am guessing someone is talking about below Bold :

Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.

Therefore, compare to cp/tar/cat, if we use rsync to copy bunch of small or big files it could cause performance issue. BUT due to i'm not able to read source code of rsync therefore i can't confirm this is the ultimate reason.

my idea is keep checking:

1. What rsync version is awenro using for testing? Could you update to latest version?
2. let see what output when use --stats and -v with --debug=FLAGS

flags

--stats This tells rsync to print a verbose set of statistics on the file transfer, allowing you to tell how effective the rsync algorithm is for your data.

--debug=FLAGS This option lets you have fine-grained control over the debug output you want to see. An individual flag name may be followed by a level number, with 0 meaning to silence that output, 1 being the default output level, and higher numbers increasing the output of that flag (for those that support higher levels). Use --debug=help to see all the available flag names, what they output, and what flag names are added for each increase in the verbose level.

at the last, i would recommend read this supplemental post to get more knowledge.
"How to transfer large amounts of data via network" moo.nac.uci.edu/~hjm/HOWTO_move_data.html