rsync: Remember copied files, even if they are deleted at the destination

Solution 1:

Two different approaches are possible:

  • make a selection of the files that need transferring and feed that list to rsync and only those files will be copied. The find command is particularly useful for that.
    For example use find -ctime 1 -print0 /path/ | rsync --files-from -
    But other sources, filename patterns, a database query, input from the application that creates the files etc. etc. are also good candidates to select specific files to copy.

  • rsync can maintain a log with the actions it performed. (Check man rsync and look for the --log-file=FILE and --log-file-format=FMT options). After a successful batch, append the file/path names from that log to the list of previously copied files. Then use that concatenated list as the --exclude-from=FILE in the next rsync run to prevent those files from getting copied again.

Note that neither approach is immediately 100% fool proof and you need to carefully consider the implications of edge cases, files that don't get copied, files that copied a second time and what happens when state/history is lost.