What is the simplest and fastest way to transfer large file through a Windows network?
I have a Window Server 2000 machine running MS SQL Server that stores over 20GB of data. The database is backed-up every day to the second harddrive. I want to transfer those backup files to another computer to build another test server and for recovery practicing. (the backup never actually got restored for almost 5 years. Don't tell my boss about that!)
I have trouble transfering that huge file through the network. I've tried plain network copy, apache download, and ftp. Any method I tried end up failing when the amount of data transfered reach 2GB. The last time that I successfully transfered the file, it was through a usb attached external harddrive. But I want to perform this task routinely and preferably automatically.
Wonder what is the most pragmatic approach for this situation ?
Solution 1:
A failure predictable at 2Gb sounds like the target filesystem is to blame... Are both on NTFS? Are you piping through any compression (zip used to fail at 2gb boundaries) ((is apache doing compression))
I have copied many files over 20Gb using robocopy (as others have mentioned) but I'd avoid using the /MIR switch until you are sure you have got the copy doing what you want - since it will delete files as well as copy them.
SMB suffers from a one packet at a time limit so is often the slower way to copy files - you have the option to copy using push or pull. Personally, I prefer the push method (copy is initiated by the source).
Solution 2:
The MS Exchange tool eseutil is an excellent utility to copy large files quickly across a network:
eseutil /y source_file /d dest_file.
Solution 3:
I highly recommend using the free utility RichCopy. It is multithreaded and can pause and resume file copy operations. I have had very good luck using it to transfer files between severs.
My three top tips for using RichCopy
If you are copying one or a few big files, set ‘File Copy’ attribute to more than '1’. It uses up resources but copies big files down quicker
If you are copying lots of files then set the ‘Thread number’ attributes to 10-10-1. This will copy multiple files quicker
If you are copying over a dodgy connection. You can re run the download and it will go and find the files it didn't manage to get the first time.
http://blogs.technet.com/markdea/archive/2009/03/24/richcopy-is-it-the-new-sliced-bread.aspx
Solution 4:
As far as file copy utilities go, TeraCopy is a nice GUI-based one (not command line) that can queue lots of files, supports pausing and resuming, can dynamically change its buffer size to optimize speed, and can optionally replace Windows Explorer's default copy/move with its own.
Solution 5:
Robocopy with the /MIR option is very useful for quick and dirty backups between machines. You can find robocopy in the Windows Server 200X Resouce Kit
MIR will MIRror the contents of one directory to another server. It will only copy files that have changed.