How can I image a hard disk multiple times simultaneously?

I want to create a image of an external USB hard disk (ext2 partition) which has large data (50 – 70 GB) to an internal IDE/SATA disk first, then restore that image file to multiple USB hard disk simultaneously in possible fastest way.

Is there any application or script available already or any suggestions?

Windows is also fine, I can mount these drives using IFS.

Solution 1:

You can use tee to output to multiple files, and bash's process substitution to redirect what would go to a file to dd which can write direct to a block device.

To copy from /dev/sda to make an image of it in a file:

dd if=/dev/sda of=./drive.image

And to copy that image to many drives:

cat drive.image | tee >(dd of=/dev/sda) >(dd of=/dev/sdb) >(dd of=/dev/sdc) | dd of=/dev/sdd

Note the last item is a pipe not a redirect, as tee will also send its input to starndard output. You could send this to /dev/null if you want to keep all the redirects the same on the command line though this is obviously a little less efficient as you running an extra process and sending an extra copy of all the info to it via a pipe only to discard it:

cat drive.image | tee >(dd of=/dev/sda) >(dd of=/dev/sdb) >(dd of=/dev/sdc) >(dd of=/dev/sdd) | dd of=/dev/null

though the performance effect is likely to be small compared to the overall operation acting on many USB drives.

You could also copy one drive direct to many other, skipping the image file, though having the image file is more efficient if you will be wanting to make more copies later (or are wanting to make more copies now than you can do in one go due to having a limited number or ports):

cat /dev/sda | tee >(dd of=/dev/sdb) >(dd of=/dev/sdc) >(dd of=/dev/sdd) | dd of=/dev/sde

You may also want to replace cat with pv (you'll find this in most standard repositories if you don't already have it installed, otherwise you can get it from here and compile it yourself) in the above to get a progress meter displayed while the copy operation is ongoing, like so:

pv drive.image | tee >(dd of=/dev/sda) >(dd of=/dev/sdb) >(dd of=/dev/sdc) >(dd of=/dev/sdd) | dd of=/dev/null

An important thing to note: depending on how your USB drives are connected and how fast they individually can write data you might find this is only a little faster than copying to each drive individually. Bandwidth on a USB bus or hub is shared between the devices connected to it, so unless you have multiple USB controllers that will be come your main bottleneck with multiple drives going (unless each drive is slow at writing, of course, as many flash based USB sticks are). Some motherboards effectively have multiple USB controllers so even without an extra controller it might be that not all your ports share the same bus, but if you are running them all through a hub they will definitely all be sharing the same I/O bandwidth.
Having said that, even if the process is little or no faster at all in terms of wall-clock time due to bus contention you still get the advantage of doing it in a single command (with a single progress bar if using pv).