Dell R730xd - Best way to configure storage for simultaneous use with ZFS & Hardware RAID

Solution 1:

Regarding your question about splitting the backplane across controllers, yes, it's possible. I've done this with 8-bay enclosures where two 4-lane SAS SFF-8087 connectors were present. 4 disks to one controller, 4 disks to the other. I don't know the composition or expander situation on the Dell server you're describing, though.

The only controllers that would allow this are those that have Mixed/Hybrid mode ports. HPE Smart Array controllers on Gen10 systems are a good example. They allow hardware RAID or HBA mode on a per-port/per drive basis.

I also have this running on an Avago MR9363-4i. /dev/sdy at the bottom is a RAID1 Hardware RAID1 for the OS. The remaining disks are split across two normal ZFS pools.

[0:0:4:0]    enclosu LSI      SAS3x40          0601  -
[0:0:5:0]    disk    SEAGATE  ST1800MM0128     E002  /dev/sda
[0:0:6:0]    disk    SEAGATE  ST1800MM0128     E002  /dev/sdb
[0:0:7:0]    disk    SEAGATE  ST1800MM0128     E002  /dev/sdc
[0:0:8:0]    disk    SEAGATE  ST1800MM0128     E002  /dev/sdd
[0:0:9:0]    disk    SEAGATE  ST1800MM0128     E002  /dev/sde
[0:0:10:0]   disk    SEAGATE  ST1800MM0128     E002  /dev/sdf
[0:0:11:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdg
[0:0:12:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdh
[0:0:13:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdi
[0:0:14:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdj
[0:0:15:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdk
[0:0:16:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdl
[0:0:17:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdm
[0:0:18:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdn
[0:0:19:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdo
[0:0:20:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdp
[0:0:21:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdq
[0:0:23:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdr
[0:0:24:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sds
[0:0:25:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdt
[0:0:26:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdu
[0:0:27:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdv
[0:0:28:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdw
[0:0:29:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdx
[0:2:0:0]    disk    AVAGO    MR9363-4i        4.68  /dev/sdy

Solution 2:

Replying to each questions:

1. while you can install another PCI-E RAID card, the backplane can be connected to a single card. So you can't concurrently use the native RAID card and the additional one to access the disks on the same internal backplane [edit: @ewwhite suggested splitting disks between controller should be possible, so maybe I am wrong here. However the only officially supported dual-controller setup on the R730xd requires the use of flex-zoning backplane option, so I am not sure if/how two controllers can be put into operation in that specific case];

2. probably not, but see #1

3. PERC H730 supports two operating modes: RAID mode and HBA mode

   • the standard RAID mode, which supports RAID and non-RAID disks. In RAID mode, the controller expect the disks to be part of a RAID array and will not expose the raw disk to the operating system. In non-RAID (or passthrough) mode, the disks declared as non-RAID are exposed to the operating system (see note #1). In this mode, SMART reporting is firmware dependent and, if not working, you can try using the dedicated megaraid driver - ie: issuing something as smartctl -a -d megaraid,0 /dev/sda;

   • in HBA mode, the card works as a normal LSI-based SAS controller. This means any RAID capabiilty is disabled and disks are directly managed by the operating system. In such a mode SMART monitoring by the card is disabled and SMART data should be directly accessible (and monitored) by the operationg system. Additionally, you lose the card LED management and you may lose the added benefit offered by the controller writeback cache (I don't have direct confirmation), meaning you get much lower performance for random writes. However, please note that in ZFS you should use a SLOG to get high randon write performance rather than relying on the controller cache anyway.

In the end, as PERC 9/10 series cards are quite flexible you have no reason to use an additional RAID card.

Note #1, from the card manual:

By default, all the disks are in RAID capable unconfigured state. The user can also convert the RAID capable disks to non-RAID disks using either the BIOS configuration utility or the UEFI/HII RAID configuration utility