ZFS on enterprise RAID pass-through, and ZFS on FreeBSD root
We've been running ZFS on top of a single hardware RAID on dell poweredge for years. I know most people are against this, but the snapshot/clone, compression and flexible partition of ZFS served us very well. Whenever a drive dies, a dell technician is dispatched (the server is in another State), he will confirm that drive has an amber light, and replace it.
Now we want to take advantage of L2ARC/ZIL caching of ZFS, we are seriously thinking about running ZFS on bare disks. Current dell RAID controllers (PERC H730/H330) do support pass-through. My remaining questions are:
if a drive fails from ZFS, does it display amber light on the front panel? This is important because, the dispatched dell technician may need to confirm the drive is indeed faulty. Otherwise we may have problem with dell.
do any people run ZFS on FreeBSD root? It is in production quality? Any known issues?
Solution 1:
You can control the PERC H730 and H330 using the LSI MegaCLI utility as both of these cards are Dell PERC badged LSI cards.
There is an excellent article and tutorial on how to do this at https://calomel.org/megacli_lsi_commands.html
I know that zfsonlinux has a ZFS Event Daemon (ZED) which you can use to cause particular things to happen on certain events (e.g. use MegaCLI to turn on the amber light for a particular slot when a drive dies).
IIRC, FreeBSD has a ZFSd which can do similar things but I am not an expert on FreeBSD so can not point you to more information other than to say that the FreeBSD forums are full of useful advice and helpful people.
I suspect that the hardest part of doing this will be figuring out what the MegaCLI "slot" number is for a given drive, because ZFS only knows about the device node / name, and doesn't have specific LSI or PERC information. if the device node name is directly related to the card and slot number, it may be a trivial transformation....otherwise, it may be quite difficult.
Even if you have to manually use MegaCLI to turn on the amber light from the shell when a drive dies to satisfy the Dell tech's procedural expectations, you're still better off giving ZFS raw drives rather than overlaying ZFS on top of hardware raid - you're losing most of the important features of ZFS by doing that, and they're the most important features (e.g. error detection and correction for your data).
Solution 2:
The DELL PERC H330 or H730 aren't suitable cards for ZFS in FreeBSD. There's a lot of misunderstandings about the "passthrough" mode of those cards and they simply don't implement this the way it should be for FreeBSD. It sure works on Windows but it's not the case with FreeBSD.
If you try to use those controllers on FreeBSD 10.2 for example, it will load with the mfi(4) driver which isn't actually the supported way to give raw disks for ZFS. For example: using this driver the SMART info will be unavailable to the operating system and this will compromise the reliability of your array.
There's workings on the mrsas(4) driver, but this driver appears to be unreliable at this moment and unusable for anything. Disk drops isn't viable on a storage system. There are some reports of this behavior overhere: https://bugs.freenas.org/issues/11764
So my recommendation at this moment is to stick with your setup. I know, this sucks. But this is the best option with ZFS at this time. Keep in mind that ZFS is really temperamental with the hardware you give to him. So it will expect a proper hardware to work as it should.
If you really want to use ZFS with it's power, get a proper HBA card (or a RAID controller that can be flashed with IT - Initiator-target - firmware) and you're good to go.
Although the PERC H330 is based on the LSI/Avago SAS3008 chipset and it's basically a clone of the LSI/Avaga 9300-8i HBA it's not possible to crossflash the H330 to IT mode, and the stock DELL firmware does not implement the proper (needed) IT mode for FreeBSD.
At this moment I'm with a similar issue. I don't care about the leds on the system or whatever, but I'm stuck with a PowerEdge R730 with one H330 and unable to proper run a ZFS storage on the machine.
Solution 3:
I was able to figure out how to do it. Wish I had thought to do it sooner.
I flashed a DELL H330 RAID card to HBA IT Firmware.
See Here: https://forums.servethehome.com/index.php?threads/crossflash-dell-h330-raid-card-to-12gbps-hba-it-firmware.25498/