What is the best way to copy 9TB of data to multiple external drives, sequentially?

The data is currently stored in a 9TB RAID array, with many users having many files and directories. We'd like to back it up to five 2TB external hard drives, but since we only have one free USB port, this must be done sequentially. This means that once hard drive #2 is plugged in, #1 is unplugged, and we won't know what was originally on hard drive #1 unless the software remembers it.

One solution would be to assign every file or directory to a hard drive beforehand, and then follow through with this as they are connected. Is there a straightforward way to do this?


Most commercial backup-software packages have support for a "removable media" option, which is what you're talking about here. It'll do a backup, prompt for operator assistance (swap drives) and continue the backup. These packages maintain a state database to keep track of which data is stored on what media, so you don't have to.

If you're in UNIX land, ye olde tar has media management options. Spanning big-big data across multiple tapes is kind of what it was built for.

If you're using XFS on Unix, xfsdump has media-management in it as well, above and beyond what tar can do.


http://wiki.bacula.org/doku.php?id=removable_disk

This document describes how to utilize removable disk drives as backup media for a backup solution using Bacula. Having seen many people ask about the use of USB disk drives as a backup media on the Bacula User's e-mail list, I felt that producing this document based on my experience with using USB disk drives as backup media might be beneficial to some bacula users.

What follows is the description of a method to use several USB disk drives as the backup media for a backup solution using Bacula and is based on my experience with setting up a backup system for a small company with three servers and around 10 workstations. This method involves emulating a traditional magazine-based tape library device by using partitions on the USB disk drives as the “magazines”. As such, the method would work equally well for disk drives in Firewire, eSATA, or any other hot-swappable enclosures supported by the OS.

I think this may be an example of what you are looking for. As stated before, any backup suite will have this option.

If you're limited to rsync then I'm guessing that manually preparing lists of files for each drive would be an option.