Storage Space Write cache, and write speed
I have been working on my NAS that is using server 2012r2 with storage spaces.
- the drives are off the shelf 7200 rpm drives
- the drives are bitlocker encrypted
- the drives are in a 8 disk parity layout
My question is this. Using SMB multichannel I have been able to transfer a 500 MB file in about 2.5 seconds at about 200MB a second to my NAS, but I see that the NAS copies that file to disk over the next 10 seconds. Why does it take that long to copy to disk? Surely 8 disks in a storage space could write more then 20-40MB a second...
Network Utilization on top, Disk Utilization on bottom
The problem comes when I try to write huge files 3+ GB to the server as the transfer speeds will go from 200MB/s down to almost 0 while it writes to disk everything that has been cached.
Solution 1:
This postponed write happens because of a a) file system level write-back RAM cache, and b) parity Spaces design, which absorbs all the writes to the internal log initially to avoid “read-modify-write” on the “critical path” dramatic slowdown. When your write is complete file system keeps flushing data in the “lazy writer” worker thread and Spaces keep reading the internal log and doing actual writes in the background as well.
You can speed the process up by a) providing dedicated flash devices for Spaces log, and b) avoiding parity and going with mirror-stripe (journaling flash will help here as well). See:
https://mrickert.com/2016/04/25/adding-journal-drives-to-windows-storage-spaces-2012/
Another option is to upgrade to Windows Server at least 2016 (2019 is recommended due to tons of Spaces related bugs fixed there) and use newly represented (Rebranded? Code is there for years...) Mirror-accelerated-Parity design. See:
http://knowledgebase.45drives.com/kb/kb450193-creating-mirror-accelerated-parity-volumes-and-storage-tiers-in-storage-spaces-windows-server-2019/
BTW, it’s bad idea to use file copy to measure your storage performance, see here why:
https://blogs.technet.microsoft.com/josebda/2014/10/13/diskspd-powershell-and-storage-performance-measuring-iops-throughput-and-latency-for-both-local-disks-and-smb-file-shares/
Good luck! :)