Why mdadm's RAID-1 is resyncing from scratch despite of the bitmap?

# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 nbd0[3] sda10[0]
      53246315 blocks super 1.2 [3/1] [U__]
      [>....................]  recovery =  1.0% (537088/53246315) finish=203.0min speed=4326K/sec
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>

# nbd-client -d /dev/nbd0
Disconnecting: que, disconnect, sock, done

# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 nbd0[3](F) sda10[0]
      53246315 blocks super 1.2 [3/1] [U__]
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>

# mdadm /dev/md/raidy --remove /dev/nbd0
mdadm: hot removed /dev/nbd0 from /dev/md/raidy

# nbd-client 10.99.99.250 7777 /dev/nbd0
Negotiation: ..size = 53247411KB
bs=1024, sz=53247411

# mdadm --incremental --run /dev/nbd0
mdadm: /dev/nbd0 attached to /dev/md/raidy which is already active.

# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 nbd0[3] sda10[0]
      53246315 blocks super 1.2 [3/1] [U__]
      [>....................]  recovery =  0.0% (31616/53246315) finish=196.2min speed=4516K/sec
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>

# uname -a
Linux vi-notebook 2.6.35-zen2-08220-g2c56b9e #14 ZEN PREEMPT Thu Oct 21 02:48:18 EEST 2010 i686 GNU/Linux

# mdadm --version
mdadm - v3.1.4 - 31

How to properly disconnect and reconnect the device to RAID-1 to take advantage of the write-intent bitmap?

Experimenting again:

Personalities : [raid1] 
md0 : active raid1 nbd0[3] sda10[0]
      53246315 blocks super 1.2 [3/2] [UU_]
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>

    /dev/md/raidy:
        Version : 1.2
  Creation Time : Tue Mar 30 05:42:53 2010
     Raid Level : raid1
     Array Size : 53246315 (50.78 GiB 54.52 GB)
  Used Dev Size : 53246315 (50.78 GiB 54.52 GB)
   Raid Devices : 3
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Mon Jan 31 18:18:03 2011
          State : active, degraded
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : vi-notebook:0  (local to host vi-notebook)
           UUID : bc325b24:fa0a4957:47820c56:fc818fa3
         Events : 2661551

    Number   Major   Minor   RaidDevice State
       0       8       10        0      active sync   /dev/sda10
       3      43        0        1      active sync   /dev/nbd0
       2       0        0        2      removed

Now removing one of the devices:

# mdadm /dev/md/raidy --fail /dev/nbd0 
mdadm: set /dev/nbd0 faulty in /dev/md/raidy
# mdadm /dev/md/raidy --remove /dev/nbd0 
mdadm: hot removed /dev/nbd0 from /dev/md/raidy

Now re-adding it:

mdadm --incremental --run /dev/nbd0

It starts resyncing from the beginning:

Personalities : [raid1] 
md0 : active raid1 nbd0[3] sda10[0]
      53246315 blocks super 1.2 [3/1] [U__]
      [>....................]  recovery =  0.4% (244480/53246315) finish=289.5min speed=3050K/sec
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>


/dev/md/raidy:
        Version : 1.2
  Creation Time : Tue Mar 30 05:42:53 2010
     Raid Level : raid1
     Array Size : 53246315 (50.78 GiB 54.52 GB)
  Used Dev Size : 53246315 (50.78 GiB 54.52 GB)
   Raid Devices : 3
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Mon Jan 31 18:22:07 2011
          State : active, degraded, recovering
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

 Rebuild Status : 0% complete

           Name : vi-notebook:0  (local to host vi-notebook)
           UUID : bc325b24:fa0a4957:47820c56:fc818fa3
         Events : 2661666

    Number   Major   Minor   RaidDevice State
       0       8       10        0      active sync   /dev/sda10
       3      43        0        1      spare rebuilding   /dev/nbd0
       2       0        0        2      removed

Hrm. Looking at your above outputs, it isn't clear that you had multiple sync'ed disks in the first place. It looks like there was a failed disk that was syncing, which was removed, then re-added, then was resyncing again. At no point do I ever see a display that shows 2 disks fully sync'ed.

I would make sure sure both disks are active, let them fully sync up together, and only after verifying that would I attempt to remove a disk and re-add it.

Note that the write intent bitmap is only an aid for quickly resyncing two disks which are already nearly in sync (i.e. after a system crash, or when a disk is removed for a short period of time). It isn't a intended for long term removal, or for disks that aren't already fully sync'ed.

You should use "--re-add" to add the removed disk back, like this:

# mdadm /dev/md0 --re-add /dev/sdf2

I just tried it, and it worked, without a rebuild - if the disk has been removed beforehand using "--remove", like you did.

Note that this is important - the disk has to have been removed using "--remove". If you just pull a disk out hard, plug it back in and try to "--re-add" it, you will get

mdadm: --re-add for /dev/sdf2 to /dev/md0 is not possible

Why is that? Looking at http://linux.die.net/man/8/mdadm, section "--re-add":

If [...] the slot that it used is still vacant, then the device will be added back to the array in the same position.

If you just pulled the disk out, the slot will still be occupied by the failed disk (marked F in /proc/mdstat):

$ cat /proc/mdstat
Personalities : [raid1] 
md0 : active raid1 loop0[0] loop3[2](F)
      1047552 blocks super 1.2 [2/1] [U_]
      bitmap: 1/1 pages [4KB], 65536KB chunk

Remove the failed disk and the "--re-add" will work, taking advantage of the bitmap.