How do I quickly copy 900,000 files?

I need to copy some data that includes 900,000 tiny files totaling around 30 gigabytes to a windows computer. However, it needs to be able to be copied and set up in under an hour and a half, and it can take 5+ hours to copy off a USB hard drive mostly due to the crazy amount of files. Is there a better/faster way to deal with this, such as doing some sort of block copying? Thanks


With large number of files, it is best to create tar archive so that you have less files to deal with. If you are using USB 2.0 external hard drive you should look at using USB 3.0, eSATA, or use a fast local network.

What is your source OS? If both of your operating systems were Linux, you could pipe the files through tar, gzip, and ssh to the target machine. You could install cygwin on Windows to get this type of functionality for Windows as well.

Update:
"Using Tar and SSH to improve SCP Speeds" post describes the commands needed to send tar contents over ssh.


If the hard drive can be removed from the USB interface and onto SATA/ATA, I would install it in the destination computer. You'll get much faster transfer speeds, as others have noted. For copying, assuming you're on Windows, I would do a simple ROBOCOPY. It's about as fast as you can really hope for, though there are other alternatives.

ROBOCOPY /E /B /MT 'source' 'destination'

I'd try to avoid compressing all these files though, there is a good chance the time to compress and move would exceed simply moving the files.

*Added the /MT option for robocopy. It can greatly speed up transfers when you're in a multi-thread environment.


You could try taking an image of the entire folder/drive

On Linux systems you can use dd to get a raw copy of the filesystem and copy it as a single large file.

To extract the image onto Windows you may need to install cygwin or a program that is able to process dd images.