How would I speed up a full disk dd?

Solution 1:

In my experience, I don't think there is something faster in the command line as dd. Adjusting the bs parameter can increase the speed, for example, I have 2 HDD that I know have a read/write speed greater than 100 MB/s so I do this:

dd if=/dev/sda of=/dev/sdb bs=100M

There is also pv (Needs to be installed first) that checks for the fastest speed on both drives and then proceeds on cloning. This has to be done of course from root:

pv < /dev/sda > /dev/sdb

With PV I got 156 MB/s

The nice thing about pv apart from the speed is that it shows the progress, current speed, time since it began and ETA. In regards to HFS+ I would not know, am just trying to help on the "speed" part. With pv or a very optimized bs parameter, you can do a 4 TB drive in less than 7 Hours (6 Hours 50 Minutes at a current speed of 150 MB/s).

I did a couple of tests with the connection types you were using and others I had available. I was using the Asus Z87 Pro and the Intel DZ68DP. This were my results, but first we need to know that the theoretical speeds for many transfer rates (Raw speeds) are just that, theory. Doing real tests revealed they are between 40% to 80% of that raw speed. This tests can change depending on Device used, connection type, motherboard, type of connecting cable, filesystem type and more. With that in mind, this is what I got (I only tested Write speed to the Device, read is typically higher):

Connected Device  -  Connection Type  -  Speed (Write Speed)
  USB 2.0                 USB 2.0              25 MB/s
  USB 3.0                 USB 2.0              35 MB/s
  USB 3.0                 USB 3.0              73 MB/s
  eSata                   eSata                80 MB/s
  Sata 2G HDD             Sata 2G              120 MB/s
  Sata 3G HDD             Sata 2G              140 MB/s
  Sata 3G HDD             Sata 3G              190 MB/s
  Sata 2G SDD             Sata 2G              170 MB/s
  Sata 3G SDD             Sata 2G              210 MB/s
  Sata 3G SDD             Sata 3G              550 MB/s 

Solution 2:

To copy a partition wholesale, use cat instead of dd. I ran benchmarks a while ago, copying a large file rather than a partition, between two disks (on the same disk, relative timings are different):

dd bs=64M    51.3
dd bs=1M     41.8
dd bs=4k     48.5
dd bs=512    48.9
cat          41.7
cp           45.3

The conclusion from this benchmark is that the choice of block size for dd matters (but not that much), and cat automatically finds the best way to make a fast copy: dd can only slow you down. With a small block size, dd wastes time making lost of tiny reads and writes. With a large block size, one disk remains idle while the other is reading or writing. The optimal rate is achieved when one disk reads while the other disk writes.

To copy a partition, it may be faster to copy the files with cp -a. This depends on how many files there are and how much of the filesystem is free space. Copying files has an overhead that's roughly proportional to the number of files, but on the other hand copying free space wastes time.

The maximum data rate for USB2 is a little under 50 MB/s which works out to 6–7 hours to transfer 1TB. This assumes a hard disk that's fast enough to saturate the USB bus; I think the faster 7200 rpm drives can do it but 5900rpm might not be that fast (maybe they are for linear writes?).

If either disk is in use in parallel, this can slow down the copy considerably as the disk heads will need to move around.