Slow backup of data from file servers with System Center Data Protection Manager 2019
I have a virtual file server cluster of 2 VMs, where each VM is running with Windows Server 2019 (8 GB RAM, 8 vCPUs). The file servers store their data on a VHD set, which is formatted with NTFS and has deduplication enabled. There is about 14 TB of data, which only consumes around 6 TB due to deduplication.
I have a physical server (64 GB RAM, 2 CPUs à 8 cores with HT, so 32 logical cores in total) with Windows Server 2019 and Microsoft SC DPM 2019 installed. It uses local HDDs (RAID6) to store the backup data on Modern Backup Storage (MBS), formatted with ReFS. This server has a redundant 10 GbE link to the file servers in the same subnet. Protection groups, that back up entire VMs or SQL databases perform well. But the protection group that backs up the data of the file server cluster has a really bad performance.
A full synchronization of the 14 (or 6) TB takes around 70 hours! That's incredibly slow. When it starts, it has about 2-4 MBit/s throughput on the 10 GbE link (which has its full potential when I copy a large file manually). After a long phase of slowness (~1 day) it "speeds" up to 200-400 MBit/s which is still quite slow.
CPUs and RAM do not seem to be a bottleneck on any server.
I found similar problems here and elsewhere:
- DPM for File Server - awfully slow?
- MS DPM - Slow performance of consistency check
- Slow Backup Speed with DPM after few minutes backup is started
- DPM 2016 MBS Performance downward spiral
- How can we improve SCDPM?
- DPM 2016 File Server Backup SLOW
But without a working solution. How can I speed up the backup of data?
Solution 1:
So far, the following tweaks brought major improvements to the backup performance of data.
Windows Defender Settings
Default Windows Defender settings will slow down the backup performance. I modified the Windows Defender settings according to this article and experienced a decent boost in backup performance. The following PowerShell lines can be used to achieve, what is described in this article.
On the DPM server:
Add-MpPreference -ExclusionPath 'C:\Program Files\Microsoft System Center\DPM\DPM\XSD'
Add-MpPreference -ExclusionPath 'C:\Program Files\Microsoft System Center\DPM\DPM\Temp'
Add-MpPreference -ExclusionPath 'E:\' # Where E:\ is the backup storage drive
Add-MpPreference -ExclusionProcess 'C:\Program Files\Microsoft System Center\DPM\DPM\bin\DPMRA.exe'
Add-MpPreference -ExclusionProcess 'C:\Windows\Microsoft.NET\Framework\v4.0.30319\csc.exe'
Add-MpPreference -ExclusionProcess 'C:\Windows\Microsoft.NET\Framework64\v4.0.30319\csc.exe'
On the file servers:
Add-MpPreference -ExclusionProcess 'C:\Program Files\Microsoft Data Protection Manager\DPM\bin\DPMRA.exe'
Enable on-the-wire compression
The activation of on-the-wire compression on the protection group for the data backup also resulted in an increased backup performance. This optimization is described here.
- Right-click the protection group and select Optimize performance... from the context menu.
- In the Network tab, check the Enable on-the-wire compression checkbox.
Registry tuning
I did not test it yet, but the following articles describe registry tweaks that might boost the backup performance even more:
- FIX: Heavy memory usage in ReFS on Windows
- ReFS volume using DPM becomes unresponsive on Windows Server 2016
- How To Optimize ReFS Performance With System Center Data Protection Manager?
Tiered Storage
If you don't go full-flash, go at least with SSD tiered storage. Microsoft highly recommends to use tiered storage as can be read here and here. Do not use HDDs only, especially not HDDs with only 7200 RPM.
Choose your RAID configuration wisely
While RAID6 provides a very high availability, it has nearly the worst performance. RAID5 has a better performance and you can use RAID5 in combination with a hot spare disk to achieve the same storage capacity and nearly the same high availability as RAID6. There are even better RAID configurations, but they are also way more expensive.