mdadm RAID6, recover 2 disk failure during reshape
I was reshaping my array from 10 disks to 11 to a degraded state (drive I want to add already has data on it, but nowhere to put it). 2 drives disconnected while it was running (power issue).
Is it still possible to recovery this array?
After power cycling them I was unable to add them to the array again:
mdadm: /dev/md0 has failed so using --add cannot work and might destroy
mdadm: data on /dev/sdX1. You should stop the array and re-assemble it.
Since rebooting, I've tried:
--assemble, fails due to "faulty" disks
--assemble --force, fails:
md: sdl1 does not have a valid v1.2 superblock, not importing!
md: sdk1 does not have a valid v1.2 superblock, not importing!
md/raid:md0: not enough operational devices (3/11 failed)
md/raid:md0: failed to run raid set.`
I've been reading the RAID Recovery article, but so far not been successful.
mdadm --create --chunk=64 --size=1953512448 --assume-clean --level=6 --raid-devices=11 /dev/md0 /dev/sd{f,h,e,g,m,i,k,l,n,d}1 missing, fails:
mdadm: /dev/sdf1 is smaller than given size. 1953512256K < 1953512448K + metadata
for all drives. My argument to --size is "Used Dev Size / 2" from mdadm --examine /dev/sdf1. I've downgraded mdadm to each version down to v3.1.2 (when default metadata was changed to 1.2, I know I never specified it manually).
Removing --size, I can create the array, but not mount:
XFS (md0): Mounting V4 Filesystem XFS (md0): Log inconsistent (didn't find previous header) XFS (md0): failed to find log head XFS (md0): log mount/recovery failed: error -5 XFS (md0): log mount failed
Info
My mdadm --detail before reshape:
/dev/md0:
Version : 1.2
Creation Time : Fri Jan 27 19:20:36 2012
Raid Level : raid6
Array Size : 15628099584 (14904.12 GiB 16003.17 GB)
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Raid Devices : 10
Total Devices : 10
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Jun 17 14:16:09 2015
State : clean
Active Devices : 10
Working Devices : 10
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : ubuntu:0
UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
Events : 6037532
Number Major Minor RaidDevice State
0 8 81 0 active sync /dev/sdf1
9 8 177 1 active sync /dev/sdl1
12 8 65 2 active sync /dev/sde1
3 8 97 3 active sync /dev/sdg1
4 8 145 4 active sync /dev/sdj1
6 8 193 5 active sync /dev/sdm1
7 8 113 6 active sync /dev/sdh1
8 8 129 7 active sync /dev/sdi1
10 8 161 8 active sync /dev/sdk1
11 8 49 9 active sync /dev/sdd1
And mdadm --examine after failure and reboot with all disks visible again:
Device paths have changed as there was a hotswap disk added before reshape started
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
Name : ubuntu:0
Creation Time : Fri Jan 27 19:20:36 2012
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 329fc32d:e9cf2ff4:3aa6c9a0:500aa445
Internal Bitmap : 2 sectors from superblock
Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
Delta Devices : 1 (10->11)
Update Time : Wed Jun 17 19:46:34 2015
Checksum : 904d0c9c - correct
Events : 6039833
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 9
Array State : A.AAA...AA. ('A' == active, '.' == missing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
Name : ubuntu:0
Creation Time : Fri Jan 27 19:20:36 2012
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : e59303ea:e613013e:ef8af657:1fc6ccab
Internal Bitmap : 2 sectors from superblock
Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
Delta Devices : 1 (10->11)
Update Time : Wed Jun 17 19:46:34 2015
Checksum : b3b3f659 - correct
Events : 6039833
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 2
Array State : A.AAA...AA. ('A' == active, '.' == missing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
Name : ubuntu:0
Creation Time : Fri Jan 27 19:20:36 2012
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 6aa0f9d8:e7b0cc66:d2f2a600:ef305279
Internal Bitmap : 2 sectors from superblock
Reshape pos'n : 3148373376 (3002.52 GiB 3223.93 GB)
Delta Devices : 1 (10->11)
Update Time : Wed Jun 17 19:46:34 2015
Checksum : 3beac20c - correct
Events : 6039833
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 0
Array State : AAAAAAAAAA. ('A' == active, '.' == missing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
Name : ubuntu:0
Creation Time : Fri Jan 27 19:20:36 2012
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4b1d87a9:16027400:df71810f:3ce53c50
Internal Bitmap : 2 sectors from superblock
Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
Delta Devices : 1 (10->11)
Update Time : Wed Jun 17 19:46:34 2015
Checksum : 91a563ea - correct
Events : 6039833
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 3
Array State : A.AAA...AA. ('A' == active, '.' == missing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
Name : ubuntu:0
Creation Time : Fri Jan 27 19:20:36 2012
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
Used Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 27c8fefa:8b2b74a2:9a456d34:d1a60c20
Internal Bitmap : 2 sectors from superblock
Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
Delta Devices : 1 (10->11)
Update Time : Wed Jun 17 19:29:09 2015
Checksum : ee4ae103 - correct
Events : 6039833
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 1
Array State : AAAAAA..AA. ('A' == active, '.' == missing)
/dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
Name : ubuntu:0
Creation Time : Fri Jan 27 19:20:36 2012
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : bebc3764:9e582fe8:01de9766:2d8c452b
Internal Bitmap : 2 sectors from superblock
Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
Delta Devices : 1 (10->11)
Update Time : Wed Jun 17 19:29:09 2015
Checksum : 6632686d - correct
Events : 6039833
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 5
Array State : AAAAAA..AA. ('A' == active, '.' == missing)
/dev/sdk1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
Name : ubuntu:0
Creation Time : Fri Jan 27 19:20:36 2012
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
Used Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 986d9f31:3a74b90d:7800779e:31607539
Internal Bitmap : 2 sectors from superblock
Reshape pos'n : 3148373376 (3002.52 GiB 3223.93 GB)
Delta Devices : 1 (10->11)
Update Time : Wed Jun 17 19:24:09 2015
Checksum : de0a23b - correct
Events : 6039833
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 6
Array State : AAAAAAAAAA. ('A' == active, '.' == missing)
/dev/sdl1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
Name : ubuntu:0
Creation Time : Fri Jan 27 19:20:36 2012
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
Used Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a5f4ac69:f6bbac94:60c1b790:db2c223e
Internal Bitmap : 2 sectors from superblock
Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
Delta Devices : 1 (10->11)
Update Time : Wed Jun 17 19:28:58 2015
Checksum : c9909fb9 - correct
Events : 6039833
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 7
Array State : AAAAAA.AAA. ('A' == active, '.' == missing)
/dev/sdm1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
Name : ubuntu:0
Creation Time : Fri Jan 27 19:20:36 2012
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 938d9190:582eecf8:b9157fce:38705df2
Internal Bitmap : 2 sectors from superblock
Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
Delta Devices : 1 (10->11)
Update Time : Wed Jun 17 19:46:34 2015
Checksum : d2462ecd - correct
Events : 6039833
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 4
Array State : A.AAA...AA. ('A' == active, '.' == missing)
/dev/sdn1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
Name : ubuntu:0
Creation Time : Fri Jan 27 19:20:36 2012
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 289f68f6:f43d8a40:2203e21c:e6cff371
Internal Bitmap : 2 sectors from superblock
Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
Delta Devices : 1 (10->11)
Update Time : Wed Jun 17 19:46:34 2015
Checksum : 4db49d1a - correct
Events : 6039833
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 8
Array State : A.AAA...AA. ('A' == active, '.' == missing)
Well the good news is that all ten drives have matching Event counts at 6039833.
I think the key to success is resolving why you are seeing...
md: sdl1 does not have a valid v1.2 superblock, not importing!
md: sdk1 does not have a valid v1.2 superblock, not importing!
This is an interesting case and certainly it will take some study to figure out a solution. If you would like some help contact me via the web link in my profile.