What parameters to mdadm, to re-create md device with payload starting at 0x22000 position on backing storage?

Solution 1:

It appears, that there is no need for any tool for setting start of the RAID payload on any given offset. The problem arose from change in offset between mdadm versions above 3.00 and below. It appears, that at the moment, due to the different offsets, the mdadm from version above 3.00 (I tried 3.24) cannot cannot re-create layout made by older mdadm. One needs to use the older one.

The problem appeared in the first place from a known bug in Ubuntu 12.04 kernel, which under certain circumstances (e.g. found during distribution upgrade...) overwrites the superblocks. The solution is to use avoid kernels from 3.2.0-22.35 to 3.2.0-24.37, which incidentaly include among other, kernel used on Ubuntu 12.04 Mini-cd installer.

This is the conversation in [email protected] mailing list, in which Mr. Robin Hill gave me the solution:

Looks like you've been hit by the bug mentioned here: http://neil.brown.name/blog/20120615073245

You've got most of the required data to recreate the array. It's just the chunk size missing I think. This is probably 512K as the arrays are using 1.2 superblocks, but it may be 64K if you were using an old mdadm version.

Your data offset is 272 sectors, which means you'll need to use an older mdadm version to re-create the array. Newer versions use 2048 sectors, but it looks like version 3.0 uses 272, so grab a copy of that if possible.

Assuming your drives are numbered in the same order as previously, the create commands will be:

mdadm -C /dev/md5 -e 1.2 -l 5 -n 5 /dev/sd{b,f,c,e,d}6 \
    --assume-clean --uuid=a3945c40:73237cd6:3d61998e:8f773d03

and

mdadm -C /dev/md6 -e 1.2 -l 6 -n 5 /dev/sd{b,f,c,e,d}5 \
    --assume-clean --uuid=2a8fd2df:3ad53552:e8b80ecf:a46d93a2

Make sure you use the curly brace format to list the disks rather than square brackets - that'll ensure the order is maintained. Also make sure that --assume-clean is used to prevent rebuilding from kicking off, and losing data.

After recreating the array, run a fsck -n on the array to check whether the details are correct. If not, stop the array and try recreating it again, adding "-c 64" to set the chunk size to 64K.

Once you've got them up and fsck shows clean, you can re-add the bitmaps using mdadm -G /dev/mdX --bitmap=internal.

Do read through Neil's blog post first as there may be something I've missed.

So after booting from XUbuntu 10.04 live CD I was able to successfully re-create the array with the given commands, and get access to all my data.

It is important to note, that the old mdadm was needed only for re-creation the array. The array can be normally accessed with 12.04 live CD (using usual mdadm --assemble --scan).

If one wants to change the layout of the raid to the newer one (with the offset 2048 sectors) the path is long:

The only way I know of to update the offset location in an existing array would be to fail the disks one at a time, zeroing the superblocks, then adding them back into the array using a new mdadm. When it's added back in, it'll use the new offset location. If you have a spare disk and a new enough kernel version, you can use the pre-emptive replacement option, which will prevent the need to run without full parity available.

...In hope that this post would keep at least one system administrator upgrading from 10.04 to 12.04 from heart-attack ;-)

See also related problem on askubuntu: mdadm fails after 10.04 -> 12.04 upgrade.