Switching mdadm to an external bitmap

I've just read this in another post about improving RAID5/6 write speeds:

After increasing stripe cache & switching to external bitmap, my speeds are 160 Mb/s writes, 260 Mb/s reads. :-D

I've already found out how to increase the stripe cache and this worked pretty well but I'd like to know more about an external bitmap. I have an incredibly fast (540MB/s) RAID0 SSD that would do well if a bitmap does what I think it does but I'm still very unsure. I've only known about them as long as I've known this post.

A few questions:

• What is a bitmap (in terms of mdadm)?
• What are the advantages of an internal bitmap (over external)?
• What are the advantages of an external bitmap (over internal)?
• How do I switch between the two?

I should add that while this is a I'm-bored-let's-break-something thread, I do value the data stored on the RAID array. If doing this is going to put data at significant risk, please let me know.

Solution 1:

What's a bitmap:

A mdadm bitmap, also called a "write intent bitmap", is a mechanism to speed up RAID rebuilds after an unclean shutdown or after removing and re-adding a disk.

With a bitmap, writing data to the RAID goes like this:

• Update bitmap: Mark the RAID chunks you are about to write to as dirty.
• Write the data to the RAID.
• Update bitmap: Mark the RAID chunks that were just written as clean.

The advantage of a bitmap is that if the system goes down in the middle of a write, the rebuild needs to check only the chunks marked as dirty, rather than the whole multi-TB RAID. This can speed up the rebuild process from taking several hours to completing in just a few seconds.

The drawback is lower write performance under normal use (outside rebuilds), since mdadm does additional disk access to update the bitmap.

External vs internal:

• external: Stored as a file on a disk outside the RAID. The advantage over an internal bitmap is better write performance during normal use (outside rebuilds).
• internal: Stored as RAID metadata. The advantage over an external bitmap is that you don't need a non-RAID disk and you save a bit on configuration (the path to the bitmap).

According to the mdadm man page:

Note:  external bitmaps are only known to work on ext2 and ext3.
       Storing bitmap files on other filesystems may result in  serious
       problems.

According to a post by Neil Brown, the mdadm author, external bitmaps should work on ext4 too:

I haven't looked inside ext4 but I am fairly confident that external bitmaps 
will work properly.

HOWTO:

Bitmaps are added and removed using mdadm --grow --bitmap=XXX ..., where the XXX is one of:

• --bitmap=internal: Create an internal bitmap.
• --bitmap=/var/my_bitmap.bin: Create an external bitmap at the specified path. The path must reside outside the RAID. A bitmap=... parameter must be added to the ARRAY entry in /etc/mdadm/mdadm.conf, and the --bitmap=... parameter must be passed if you are assembling the RAID from the command line.
• --bitmap=none: Remove/disable any bitmaps.

References:

• HowTo: Speed Up Linux Software Raid Building And Re-syncing
• mdadm man page - see flag -b, --bitmap=
• mdadm.conf man page - see option bitmap

Comments:

IMO, bitmaps are perhaps primarily of interest for RAID levels 5 and 6, since these have the slowest rebuilds.

I switched from RAID 5 to RAID 10 myself; the rebuilds are so much faster that I don't feel the need for a bitmap, and RAID 10 seems to require far fewer rebuilds in the first place.

My RAID 5 setup used to drop a disk something like once a month, causing 12-14 hour rebuilds. The RAID 10 has only dropped a disk once in half a year, rebuilt in less than an hour.

I don't know if the frequent disk drops I experienced was caused by something other than the RAID level, but the RAID 10 has been far more stable and rebuild speed isn't much of a concern anymore.

Solution 2:

I have been using md RAID 5 for a while now, and not had any disk dropouts. I was interested to see what the addition of an external bitmap storage would gain me.

My current setup includes a boot disk and then 4 x 3TB disks as RAID 5. Last night I added another 3TB disk and after 20 odd hours it's 70% thru reshaping. Since reading this post, I've decided that after it's finished reshaping I will definitely add the external bitmap to the boot disk (1TB).

I have md RAID 5 base layer, LVM2 over that and some ISCS targets to some of the LVM2 partitions with various virtual machines being tested on the ISCSI.

I've done a bunch of benchmark images, posted here if you're interested.