HW RAID1 or ZFS mirror

Solution 1:

ZFS works optimally if it knows as much as possible about the underlying physical disk structure. A hardware RAID 1 will present a single volume to the OS which ZFS will then treat as a single disk. You'll therefore miss out on the benefits of error correction / fault detection / checksumming etc.

There's another benefit to using ZFS over HW RAID - if your hardware dies, be it the motherboard, RAID card, CPU etc, you can pull the disks and use them in any other system that is running ZFS, and the entire filesystem is readable. If you use HW RAID, you may have a dependency on that specific RAID card, so if it fails you might need to find a similar model before you can read the disks again. You can mitigate this by having dual RAID cards, but this usually starts to become expensive and complex.

In my experience, for the majority of use-cases that I've used a mirrored disk configuration, the performance difference between ZFS and HW RAID wasn't great. If performance is your #1 priority, HW RAID will offer the best solution. In all other cases, I believe ZFS is the better choice.

Solution 2:

Although I love ZFS if you're doing a LOT of writing then take into consideration that any software RAID 1/10 will need to send every write twice over the bus and through the controller - meaning you can get into congestion and queue issues.

Solution 3:

ZFS has lots of built in optimizations that work best when used without Raid controllers. On a ZFS Raid1 you get read performance that is usually superior to a hardware Raid, also with RaidZ you can get superior performance to Hardware Raid5 Just take a look at Anadtech.

What you absolutely need to take into consideration is how ZFS caches though. You should basically have 4GB of Ram to use it at all, 8GB to get some decent performance, but more is always good. There are lots of cases where more Ram helps with ZFS. The best ways to improve performance though are seperate devices (SSDs) for the ZIL and L2ARC, as those generally tend to be bottlenecking performance. This explains the idea behind L2ARC a little.

The principal problem with Hardware Raid here is, that ZFS does, what the Raid Controller does, so when using a Hardware Raid below ZFS, there is a Raid Controller that expects itself and only itself to be doing Queueing, Caching, IO Optimization and so on and there is ZFS on top of it also expecting the same of itself. This has side effects of course and they aren't necessarily pretty and will most likely have negative performance effects due to the way ZFS does I/O Parallelization. Also its predictive "self healing" features will in part be kind of useless that way. Therefor it is specifically recommended that when you use a Raid controller, you should expose the Disks JBOD are as Single-Disk Raid0 devices, so that ZFS can do the housekeeping by itself.