Best way to copy large amount of data between partitions

I'm looking to transfer data across 2 lv of an HP-UX server. I have a couple of those transfers to do, some of which are mostly binary (Oracle tablespace...) and some others are more text files (logs...). Used data size of the volumes is between 100Gb and 1Tb. Also, I will be changing the block size from 1K to 8K on some of these partitions...

Things I'm looking for:

• Guarantees data integrity
• Fastest data transfer speed
• Keeps file ownership and permissions

Right now, I've thought about dd, cp and rsync, but I'm not sure on the best one to use and the best way to use them...

You don't want to use dd. That is for working on 1 file or stream, not on a whole filesystem.

rsync is designed to do what you want but as the previous poster has stated, and as my tests have shown, it's not the fastest. That's because it's for doing something like this: "Ok, I'm looking at file A. Is file A on the destination? If so, is it newer, older, the same?" Etc. rsync is a bit complicated because it is meant to be run more than once... like the name says, it's for synchronizing two locations.

For doing the sort of thing you want, I have found a tar copy to be quick, easy, and reliable. Tar knows about hard links. Tar knows about devices. Tar handles almost any situation you'll run into in your filesystem (except really long paths, and, if you're not using Gnu tar, you may need to be wary of putting a / in the beginning of your pathname).

Anyway, I've had 99.98% success for the last 20 years doing by doing this:

cd /my/source; tar cf - subdirectory | (cd /destination/path; tar xf -)

...The subdirectory you want to copy will show up in /destination/path .

If you like to watch your progress, you can use "xvf" instead of "xf" in the latter portion of that string.

...my 0.02% failures have come from really long file paths... :-(

Tar will not guarantee file integrity. That said, as long as you don't see any error messages, I've found it to be very reliable. It will keep permissions and ownership properly.

Have a look at this post. Some answers suggested using tar. Others suggested using rsync. They are taking about copying data between two machines. Your problem is similar, but you need to copy the files locally instead of doing it over the network.

I would recommend using rsync, as it has features that specifically address most of your concerns. If you use appropriate options (e.g. the -a option), then all file ownerships, permissions, and times will be preserved. Furthermore, rsync automatically uses checksums to ensure that all transferred files arrive at the destination intact, so data integrity is assured (presuming a successful run).

The only point where rsync may not be optimal is speed, especially when compared to a lighter-weight alternative like cp, but I doubt that you would notice much difference, unless your processing power is very low.