ZFS: RAIDZ versus stripe with ditto blocks

I'm going to build a ZFS file server from FreeBSD. I learned recently that I can't expand a RAIDZ udev once it's part of the pool. That's a problem since I'm a home user and will probably add one disk a year tops.

But what if I set copies=3 against my entire pool and just throw individual drives into the pool separated? I've read somewheres that the copies will try and distribute across drives if possible. Is there a guarantee there? I really just want protection from bit rot and drive failure on the cheap. Speed's not an issue since it'll go over a 1Gb network and at MOST stream 720p podcasts.

Would my data be guaranteed safe from a single drive failure? Are there things I'm not considering? Any and all input is appreciated.

Solution 1:

You will not be protected using a raidz vdev + single disk vdevs in the same pool. You should be adding two disks at the time (and mirror them) to your zpool, that will cover one disk failure.

The reason is that copies does not guarantee that the blocks will end up on different disks.

One alternative to expanding your raidz vdev is to use zfs send to store all you data somewhere temporarily while you add a disk and rebuild your raidz vdev and then zfs receive to get it back. It will be hard once you get past a few TB as you need a lot of space to store your data.

Solution 2:

The file system data would very likely be redundant, especially with copies=3 so your data would be protected against bit rot.

However, it won't be reliably protected against a full disk failure as there would be no way to replace the faulty disk by a new one. Even while just after the failure your should be able to access all your data, your system won't survive a reboot as your pool would not be importable.

See ZFS: How do you restore the correct number of copies after losing a drive? for a similar question.

Solution 3:

Interesting, found this today. Copies should guarantee ditto blocks created by 'copies' are stored on different vdevs: https://blogs.oracle.com/bill/entry/ditto_blocks_the_amazing_tape (see section "Spread 'em")

So, according to this blog post, you would be protected against full disk failures on file systems which have multiple copies available. But only on those file systems.

"In a storage pool with multiple devices (vdevs), things get a little spicier. We allocate each copy of a block on a separate vdev"