rsync: Remember copied files, even if they are deleted at the destination
Solution 1:
Two different approaches are possible:
-
make a selection of the files that need transferring and feed that list to rsync and only those files will be copied. The
find
command is particularly useful for that.
For example usefind -ctime 1 -print0 /path/ | rsync --files-from -
But other sources, filename patterns, a database query, input from the application that creates the files etc. etc. are also good candidates to select specific files to copy. -
rsync can maintain a log with the actions it performed. (Check
man rsync
and look for the--log-file=FILE
and--log-file-format=FMT
options). After a successful batch, append the file/path names from that log to the list of previously copied files. Then use that concatenated list as the--exclude-from=FILE
in the next rsync run to prevent those files from getting copied again.
Note that neither approach is immediately 100% fool proof and you need to carefully consider the implications of edge cases, files that don't get copied, files that copied a second time and what happens when state/history is lost.