Upgraded Ubuntu, all drives in one zpool marked unavailable

Solution 1:

After lots and lots more Googling on this specific error message I was getting:

root@kyou:/home/matt# zpool import -f storage
cannot import 'storage': one or more devices are already in use

(Included here for posterity and search indexes) I found this:

https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/VVEwd1VFDmc

It was using the same partitions and was adding them to mdraid during any boot before ZFS was loaded.

I remembered seeing some mdadm lines in dmesg and sure enough:

root@kyou:/home/matt# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md126 : active raid5 sdd[2] sdb[0] sde[1]
      1953524992 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

These drives were, once upon a time, part of a software raid5 array. For some reason, during the upgrade, it decided to rescan the drives, and find that the drives were once part of an md array, and decided to recreate it. This was verified with:

root@kyou:/storage# mdadm --examine /dev/sd[a-z]

Those three drives showed a bunch of information. For now, stopping the array:

root@kyou:/home/matt# mdadm --stop /dev/md126
mdadm: stopped /dev/md126

And re-running import:

root@kyou:/home/matt# zpool import -f storage

has brought the array back online.

Now I make a snapshot of that pool for backup, and run mdadm --zero-superblock on them.

Solution 2:

Ubuntu seems to have some annoying udev issues that we don't see on the Red Hat/CentOS side. I'd recommend using the WWN-based device names if you can, as they seem less susceptible to this.

Have you seen: Why did rebooting cause one side of my ZFS mirror to become UNAVAIL?

Solution 3:

I ran into almost this exact problem trying to upgrade to the 3.13 series kernels on Debian Wheezy. You are right in your comment; it is a udev bug. I never did get it sorted unfortunately but it's worth exploring other kernels, especially the 3.11 series, for compatibility with the 0.6.2 version of ZOL. Just use the older kernel until 0.6.3 comes out.