Increase the size of software RAID 10

Currently I have 8 disks each of size 32G forming a RAID 10. Now, I want to increase the size of this RAID by adding extra disks. This is a production device so there is already critical data in the RAID. The filesystem is XFS. Is there any way to increase the size of this RAID without affecting the running read/writes on that RAID. If not, how to do this with minimum offline time ?


Solution 1:

The existing answers are quite outdated. Here in 2020, it's now possible to grow an mdadm software RAID 10, simply by adding 2 or more same sized disks.

Creating the example RAID 10 array

For testing purposes, instead of physical drives, I created 6x 10GB LVM volumes, /dev/vg0/rtest1 to rtest6 - which mdadm had no complaints about.

# Using the thinpool lvthin on VG vg0 - I created 6x 10G volumes
lvcreate -T vg0/lvthin -V 10G -n rtest1 
lvcreate -T vg0/lvthin -V 10G -n rtest2
...

Next, I created a RAID 10 mdadm array using the first 4 rtestX volumes

mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/vg0/rtest[1-4]

Using mdadm -D (equal to --detail), we can see the array has 4x "drives", with a capacity of 20GB out of the 40GB of volumes, as is expected with RAID 10.

root@host ~ # mdadm -D /dev/md0

/dev/md0:
           Version : 1.2
     Creation Time : Fri Nov 20 09:02:39 2020
        Raid Level : raid10
        Array Size : 20953088 (19.98 GiB 21.46 GB)
     Used Dev Size : 10476544 (9.99 GiB 10.73 GB)
      Raid Devices : 4
     Total Devices : 4
       Persistence : Superblock is persistent

       Update Time : Fri Nov 20 09:04:24 2020
             State : clean
    Active Devices : 4
   Working Devices : 4
    Failed Devices : 0
     Spare Devices : 0

            Layout : near=2
        Chunk Size : 512K

Consistency Policy : resync

              Name : someguy123:0  (local to host someguy123)
              UUID : e49ab53b:c66321f0:9a4e272e:09dc25b1
            Events : 23

    Number   Major   Minor   RaidDevice State
       0     253        9        0      active sync set-A   /dev/dm-9
       1     253       10        1      active sync set-B   /dev/dm-10
       2     253       11        2      active sync set-A   /dev/dm-11
       3     253       12        3      active sync set-B   /dev/dm-12

Expanding the RAID10 with 2 new equal sized volumes/disks

To grow the array, first you need to --add the pair(s) of disks to the array, then use --grow --raid-devices=X (where X is the new total number of disks in the RAID) to request that mdadm reshapes the RAID10 to use the 2 spare disks as part of the array.

mdadm --add /dev/md0 /dev/vg0/rtest5 /dev/vg0/rtest6
mdadm --grow /dev/md0 --raid-devices=6

Monitor the resync process

Here's the boring part - waiting for anywhere from minutes, to hours, to days or even weeks depending on how big your RAID is, until mdadm finishes reshaping around the new drives.

If we check mdadm -D - we can see the RAID is currently reshaping.

mdadm -D /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Fri Nov 20 09:02:39 2020
        Raid Level : raid10
        Array Size : 20953088 (19.98 GiB 21.46 GB)
     Used Dev Size : 10476544 (9.99 GiB 10.73 GB)
      Raid Devices : 6
     Total Devices : 6
       Persistence : Superblock is persistent

       Update Time : Fri Nov 20 09:15:05 2020
             State : clean, reshaping
    Active Devices : 6
   Working Devices : 6
    Failed Devices : 0
     Spare Devices : 0

            Layout : near=2
        Chunk Size : 512K

Consistency Policy : resync

    Reshape Status : 0% complete
     Delta Devices : 2, (4->6)

              Name : someguy123:0  (local to host someguy123)
              UUID : e49ab53b:c66321f0:9a4e272e:09dc25b1
            Events : 31

    Number   Major   Minor   RaidDevice State
       0     253        9        0      active sync set-A   /dev/dm-9
       1     253       10        1      active sync set-B   /dev/dm-10
       2     253       11        2      active sync set-A   /dev/dm-11
       3     253       12        3      active sync set-B   /dev/dm-12
       5     253       14        4      active sync set-A   /dev/dm-14
       4     253       13        5      active sync set-B   /dev/dm-13

Enjoy your larger RAID10 array!

Eventually once mdadm finishes reshaping, we can now see that the array size is ~30G instead of ~20G, meaning that the reshaping was successful and relatively painless to do :)

mdadm -D /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Fri Nov 20 09:02:39 2020
        Raid Level : raid10
        Array Size : 31429632 (29.97 GiB 32.18 GB)
     Used Dev Size : 10476544 (9.99 GiB 10.73 GB)
      Raid Devices : 6
     Total Devices : 6
       Persistence : Superblock is persistent

       Update Time : Fri Nov 20 09:25:01 2020
             State : clean
    Active Devices : 6
   Working Devices : 6
    Failed Devices : 0
     Spare Devices : 0

            Layout : near=2
        Chunk Size : 512K

Consistency Policy : resync

              Name : someguy123:0  (local to host someguy123)
              UUID : e49ab53b:c66321f0:9a4e272e:09dc25b1
            Events : 93

    Number   Major   Minor   RaidDevice State
       0     253        9        0      active sync set-A   /dev/dm-9
       1     253       10        1      active sync set-B   /dev/dm-10
       2     253       11        2      active sync set-A   /dev/dm-11
       3     253       12        3      active sync set-B   /dev/dm-12
       5     253       14        4      active sync set-A   /dev/dm-14
       4     253       13        5      active sync set-B   /dev/dm-13

Solution 2:

There are 2 cases:

  1. you add new drives The easiest and safest way is to create a new array on the new drives , create a physical volume on the new array and here you are. No loss of performance here. As a bonus you can create a new volume group in order to put your data in one array or the other.
  2. You replace existing drives with bigger ones Replace them one by one, each time create 2 partitions on the disk. You add the first (for example sdX1 to the existing array ( it should recover automatically ) and you can then create a new array on all the second partions ( the sdX2 ). Depending on your usage there might be a performance hit for some operations ( basically if you copy data between both arrays ).

In both cases you won't lose data and if your hardware allows hotplug you will not have downtime.

By the way even if mdadm allowed a dynamic resize of the array I would not take the chance with production data.