How to add disk back to RAID and replace removed

Received a message that RAID went into degraded state:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sda1[0] sdb1[1]
      524224 blocks [2/2] [UU]

md2 : active raid10 sda5[0] sdb5[1] sdd5[3] sdc5[2](F)
      1128864768 blocks super 1.2 512K chunks 2 near-copies [4/3] [UU_U]

md1 : active raid10 sda2[0] sdb2[1] sdc2[2] sdd2[3]
      41909248 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]

unused devices: <none>

Logged in to the server and removed fail disk (didn't mark it as failed):

#> mdadm --manage /dev/md2 --remove /dev/sdc5

Here is current status:

#> cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda1[0] sdb1[1]
      524224 blocks [2/2] [UU]

md2 : active raid10 sda5[0] sdb5[1] sdd5[3]
      1128864768 blocks super 1.2 512K chunks 2 near-copies [4/3] [UU_U]

md1 : active raid10 sda2[0] sdb2[1] sdc2[2] sdd2[3]
      41909248 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]

unused devices: <none>

And more details:

#> mdadm --detail /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Sun Apr  6 13:42:43 2014
     Raid Level : raid10
     Array Size : 1128864768 (1076.57 GiB 1155.96 GB)
  Used Dev Size : 564432384 (538.28 GiB 577.98 GB)
   Raid Devices : 4
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Wed Jun  3 10:29:14 2015
          State : active, degraded 
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 512K

           Name : yyc-controller:2
           UUID : c9fe371a:2c8345bf:68054eba:bb4d78aa
         Events : 41545

    Number   Major   Minor   RaidDevice State
       0       8        5        0      active sync   /dev/sda5
       1       8       21        1      active sync   /dev/sdb5
       2       0        0        2      removed
       3       8       53        3      active sync   /dev/sdd5

Now I'm trying add it back but it fails:

#> mdadm --manage /dev/md2 --add /dev/sdc5
mdadm: add new device failed for /dev/sdc5 as 4: Invalid argument

How can I add disk as Number 2 instead of removed?


I had a similar issue. One disk of a raid1 had some issues and I had removed the entire disk from the raid. After a destructive badblocks check writing random patterns, three of the four partitions of the disk could be added back to the raid without problems. Only the last partition failed with the above error. syslog reported:

md: sdb4 does not have a valid v1.2 superblock, not importing!

I didn't want to stop the raid and completely rebuild it, as sometimes suggested, but I could fix this by zeroing out the random patterns, where mdadm seemd to find some superblock like bytes using dd:

dd of=/dev/sdb4 if=/dev/zero bs=1M count=1

Afterwards mdadm could add the partition without errors.