How ZFS handles online replacement in a RAID-Z (theoretical)

Solution 1:

Testing with a file based pool (v28 on FreeBSD 8.3 using file-backed md devices) suggests that it should work. I was able to offline one of the remaining disks while the resilver was in progress. Ideally it'd need testing with real disks and to actually pull one to be 100% sure but ZFS was perfectly happy to let me offline the disk.

Before offlining md0, the pool was still fully ONLINE so it appears to me that ZFS is just mirroring the replaced disk to the new disk, but still treating the whole lot as available during the process.

    NAME                     STATE     READ WRITE CKSUM
    test                     DEGRADED     0     0     0
      raidz1-0               DEGRADED     0     0     0
        8480467682579886773  OFFLINE      0     0     0  was /dev/md0
        md1                  ONLINE       0     0     0
        replacing-2          ONLINE       0     0     0
          md2                ONLINE       0     0     0
          md3                ONLINE       0     0     0  (resilvering)

Solution 2:

Disk C is still used in the RAIDZ exactly as it had been until it is removed from the VDev. As Matt points out, ZFS replaces a disk by making the replacement disk a mirror of the replacee, and resilvering the replacement disk. The RAIDZ VDev is never degraded, and never resilvered (until A fails, which is entirely separate from the replacement operation).