ZFS: Mirror vs. RAID-Z

The simple answer is that to mirror something takes almost no processing power - it just writes to the disk a second time. For RAID-Z2, you have to compute an entirely new parity block, which although small CAN bog down the CPU when you have to write large amounts of data quickly.

Mirroring is always the preferred solution for high-speed data, if it's just bulk-storage without fast write speeds, RAID-Z2 is a good alternative that does allow any two drives to die as you allude to.

The other advantage is that mirrored pools can be expanded with more mirrored devices - while a RAID-Z2 can not be expanded - though more RAID-Z2 storage can be added to the pool, it will be two RAID-Z2 storage pools concatenated (in effect) rather than equally split between all the storage and striped.

RAID-Z eliminates most of the write penalty and the data integrity issues that RAID 5/6 volumes suffer from, at the cost of some CPU time. Typically, systems have CPU cycles to spare, so spending CPU time to improve IO performance and data integrity is a good compromise vs. mirroring.

Here is a detailed explanation of RAID-Z that may answer other questions.

Also, remember that RAID is a fault-tolerance solution. You don't implement RAID-Z2 to protect against data loss -- you perform backups or replicate to do that. You choose to implement RAID-Z2 vs. RAID-Z or RAID-10 vs RAID-6 vs. RAID-5 to keep your systems operational in the event of hardware failure.

The main performance difference between mirrors and RAIDZ1/2 [1] is not the expected CPU usage (which we have plenty these days), but the fact that ZFS random IOPS depends on the total number of vdevs rather than the total number of disk.

For example, 4x 2-way mirrors provides up to 8x the random read performance and 4x the random write performance of a single disk. On the other hand, a 8x RAIDZ2 vdev (6 data + 2 parity) is going to provide the same random IOPS of a single disk.

In other words: for random IO heavy workload, mirrors is the best choice performance wise.

[1]: RAIDZ3 is significantly heavier CPU wise.