Apparently Robocopy does not understand Windows Server 2016 deduplication. Is there a way to make it work without corrupting the volume?

The System Volume Information directory should be excluded using the /XD switch. Probably a good idea to exclude other hidden/system directories such as $RECYCLE.BIN.


Two command line switches that were used lead to this: /MIR and /ZB. As the documentation ( robocopy /??? ) describes:

/MIR :: MIRror a directory tree (equivalent to /E plus /PURGE).
/ZB :: use restartable mode; if access denied use Backup mode.

It's the combination that did you in: /MIR will delete (as pointed out when you run robocopy without arguments) and "Backup mode" defeats most permissions in order to be able to read files "normally" unreadable in order to make complete backups.

"Backup mode" is notably undefined in the "help" description. You've got to know that the Windows CreateFile API supports a flag called FILE_FLAG_BACKUP_SEMANTICS, which in combination with a certain access right SE_BACKUP_NAME (which is given to the Administrator group by default - also the Backup Operators group, duh) bypasses normal file security.

You didn't know that? Then you may also not know that robocopy wasn't originally part of Windows at all - it was part of a supplement called the "Windows Resource Kit" which was used mainly by programmers and hard-core sysadmins back in the day, and although it was grandfathered into the Windows distribution back in Windows Server 2008 it has never ever received any attention - except for additional performance options, woot! Particularly, no attention from program managers dedicated to UI or usability. So it's a raw bit of power that can be used - or misued! - at your own risk.

(A good rule of thumb: Don't use command line options you don't really understand.)

Information you might like to know about "Backup mode" file access:

https://isc.sans.edu/forums/diary/Use+The+Privilege/20483/

https://docs.microsoft.com/en-us/windows/desktop/api/FileAPI/nf-fileapi-createfilea

https://docs.microsoft.com/en-us/windows/desktop/FileIO/file-security-and-access-rights


Here are the followup results using the other answers provided, and testing with a deduplicated destination. (Meta: I don't know if I should be including this as an edit at the bottom of my original question.)

The Robocopy command line evolved to finally look like this:

robocopy \\OLD-SERVER\e$\ \\NEW-SERVER\e$\ /MIR /COPYALL /DCOPY:DAT /NP /Z /B /J /SL /MT:128 /R:1 /W:10 /LOG+:robocopy-log.txt /TEE /XD "Recycler" "Recycled" "$Recycle.bin" "System Volume Information" /XF "pagefile.sys" "swapfile.sys" "hiberfil.sys"

Options and purpose:

  • /MIR - Mirror source to destination, and delete files and directories on the destination, if they are no longer present on the source
  • /COPYALL - Copy all file info: data, attributes, and timestamps, NTFS Security ACLs, Owner info, Auditing info (not all included by default)
  • /DCOPY:DAT - Copy all directory info - data, attributes, timestamps (original creation timestamp is not copied by default; normally this changes to the date that it was copied by Robocopy)
  • /NP - Don't display progress
  • /Z - Use restartable mode
  • /B - Copy files in Backup mode (I don't know if this is needed for user directories where they are the exclusive owner, excluding the administrator. This option will destroy a deduplicated destination volume without excluding "System Volume Information")
  • /J - Copy using unbuffered I/O (faster copy of large multi-gig files)
  • /SL - Copy symbolic links rather than the target
  • /MT:128 - Use maximum CPU threads (better use of 10 gigabit Ethernet and many CPU cores)
  • /R:1 - If file access error, retry 1 time
  • /W:10 - If file access error, wait 10 seconds before retry
  • /LOG+ - Log the output to text file, append if log file already exists
  • /TEE - Print results to screen and to log file
  • /XD - Exclude directories, and everything within them. Names with spaces in them need be enclosed in quotes: "Recycler" "Recycled" "$Recycle.bin" "System Volume Information"
  • /XF - Exclude files: virtual memory and hibernation files if they happen to be present on the source: "pagefile.sys" "swapfile.sys" "hiberfil.sys"

Final re-run:

            Total    Copied   Skipped  Mismatch    FAILED    Extras 
 Dirs :    158189    153466    158186         0         0         0
Files :   1116292         0   1116296         0         0         0
Bytes :   1.350 t         0   1.350 t         0         0         0
Times :   0:01:04   0:00:00                       0:00:00   0:01:04

Deuplication report

,

Also, I do not know the proper channels to report bugs to Microsoft, but I have linked to this discussion at the bottom of Microsoft's deduplication documentation, on their Windows IT Pro Center website:

https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/overview