Ubuntu 16.04 - frozen mdadm array

Solution 1:

Using LVM for this was indeed a mistake. Not only does it make an unnecessary complicated storage stack for anyone other than its creator, MD arrays are built before LVM arrays, requiring you to manually invoke MD scanning on your LVs that are acting as MD members.

Additionally, avoid the use of kernel device names in persistent configurations (such as sda, sdb, etc). This is especially relevant when naming a volume group, as VGs abstract away underlying storage and can be freely moved across PVs. Kernel device names are also not considered to be permanent, and can change at any time for various reasons. This isn't a problem for LVM PVs (as they're part of a wholesale disk scan and will pick up just about anything), but your VG name will quickly not reflect reality in the situation you've created.

I would recommend that you attempt to gracefully remove the LV from your MD array, and bring it back to a degraded (but sane) state. Be aware that MD on top of LVM isn't something people care about when bug smashing. You're in uncharted territory, and things that you expect to work may fail for no apparent reason.

If this data is critical and not backed up, you way want to defer to someone on-site that knows LVM and MD really really well. I'm assuming you don't have that since you're asking here, so let's have a conversation if you need it. I'll update this with any interesting details if you have to go that route. For now, just try to backpedal by replacing the LVM mess with a plain old disk for a member.