How can I use rsync to duplicate a directory tree, creating hardlinks to files?
From time to time, I have to perform several large migration changes on data files on my server, and I'm looking for a good way to do this. I was thinking about using rsync to duplicate my directory structure starting at the root data folder, creating hard links to all original the files (some of them are rather big), and I can overwrite in the destination tree only the files that need migrating. In the end, I can safely switch from the old files to the new files with two mv operations.
However, I can't seem to get rsync to do this. I tried
rsync -a --link-dest=$DATA $DATA $DATA/../upgrade_tmp
but instead of creating hard links to files, rsync copies them entirely. Is there a problem using the same source and link-dest directory?
Solution 1:
rsync is a powerful tool, but it is, unfortunately, strangely picky about some of its pathnames.
If $DATA is an absolute path (i.e. it begins with a /), then the correct command-line to use is:
rsync -a --link-dest=$DATA $DATA/ $DATA/../upgrade_tmp
[Now, just a brief aside about rsync's strangeness. Note the trailing / added to the source argument. This tells rsync to work with the contents of the source directory, rather than with the source directory itself. (I'm assuming that $DATA doesn't already contain a trailing /.) In this case, we want to work with the contents, so we add the trailing /.]
If, on the other hand, $DATA is a relative path (i.e. it does not begin with a /), then Sean R's comment about --link-dest is bang on: The link-dest path is interpreted relative to the destination path, so you would use the following:
rsync -a --link-dest=../`basename $DATA` $DATA/ $DATA/../upgrade_tmp
EDIT
One final note, it turns out that the second rsync command-line I gave should work regardless of whether $DATA is an absolute path, since basename doesn't care whether a path is absolute or relative.
Solution 2:
What you want is "cp -al":
cp -al $DATA/ $DATA/../upgrade_tmp/
- -a recurses like
rsync -a - -l will hard link files instead of copying them.
Solution 3:
Turns out it is more difficult to do this with rsync than with other tools. The correct answer for rsync is Steven Monai's, but the easiest way to do this is to use either cp -al or pax -rwl on systems where -l is not a valid option for cp:
pax -rwl $DATA $DATA/../upgrade_tmp
or
cp -al $DATA/ $DATA/../upgrade_tmp/