Increase the size of software RAID 10
Currently I have 8 disks each of size 32G forming a RAID 10. Now, I want to increase the size of this RAID by adding extra disks. This is a production device so there is already critical data in the RAID. The filesystem is XFS. Is there any way to increase the size of this RAID without affecting the running read/writes on that RAID. If not, how to do this with minimum offline time ?
Solution 1:
The existing answers are quite outdated. Here in 2020, it's now possible to grow an mdadm software RAID 10, simply by adding 2 or more same sized disks.
Creating the example RAID 10 array
For testing purposes, instead of physical drives, I created 6x 10GB LVM volumes, /dev/vg0/rtest1 to rtest6 - which mdadm had no complaints about.
# Using the thinpool lvthin on VG vg0 - I created 6x 10G volumes
lvcreate -T vg0/lvthin -V 10G -n rtest1
lvcreate -T vg0/lvthin -V 10G -n rtest2
...
Next, I created a RAID 10 mdadm array using the first 4 rtestX volumes
mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/vg0/rtest[1-4]
Using mdadm -D (equal to --detail), we can see the array has 4x "drives", with a capacity of 20GB out of the 40GB of volumes, as is expected with RAID 10.
root@host ~ # mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Fri Nov 20 09:02:39 2020
Raid Level : raid10
Array Size : 20953088 (19.98 GiB 21.46 GB)
Used Dev Size : 10476544 (9.99 GiB 10.73 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Fri Nov 20 09:04:24 2020
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 512K
Consistency Policy : resync
Name : someguy123:0 (local to host someguy123)
UUID : e49ab53b:c66321f0:9a4e272e:09dc25b1
Events : 23
Number Major Minor RaidDevice State
0 253 9 0 active sync set-A /dev/dm-9
1 253 10 1 active sync set-B /dev/dm-10
2 253 11 2 active sync set-A /dev/dm-11
3 253 12 3 active sync set-B /dev/dm-12
Expanding the RAID10 with 2 new equal sized volumes/disks
To grow the array, first you need to --add the pair(s) of disks to the array, then use --grow --raid-devices=X (where X is the new total number of disks in the RAID) to request that mdadm reshapes the RAID10 to use the 2 spare disks as part of the array.
mdadm --add /dev/md0 /dev/vg0/rtest5 /dev/vg0/rtest6
mdadm --grow /dev/md0 --raid-devices=6
Monitor the resync process
Here's the boring part - waiting for anywhere from minutes, to hours, to days or even weeks depending on how big your RAID is, until mdadm finishes reshaping around the new drives.
If we check mdadm -D - we can see the RAID is currently reshaping.
mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Fri Nov 20 09:02:39 2020
Raid Level : raid10
Array Size : 20953088 (19.98 GiB 21.46 GB)
Used Dev Size : 10476544 (9.99 GiB 10.73 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent
Update Time : Fri Nov 20 09:15:05 2020
State : clean, reshaping
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 512K
Consistency Policy : resync
Reshape Status : 0% complete
Delta Devices : 2, (4->6)
Name : someguy123:0 (local to host someguy123)
UUID : e49ab53b:c66321f0:9a4e272e:09dc25b1
Events : 31
Number Major Minor RaidDevice State
0 253 9 0 active sync set-A /dev/dm-9
1 253 10 1 active sync set-B /dev/dm-10
2 253 11 2 active sync set-A /dev/dm-11
3 253 12 3 active sync set-B /dev/dm-12
5 253 14 4 active sync set-A /dev/dm-14
4 253 13 5 active sync set-B /dev/dm-13
Enjoy your larger RAID10 array!
Eventually once mdadm finishes reshaping, we can now see that the array size is ~30G instead of ~20G, meaning that the reshaping was successful and relatively painless to do :)
mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Fri Nov 20 09:02:39 2020
Raid Level : raid10
Array Size : 31429632 (29.97 GiB 32.18 GB)
Used Dev Size : 10476544 (9.99 GiB 10.73 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent
Update Time : Fri Nov 20 09:25:01 2020
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 512K
Consistency Policy : resync
Name : someguy123:0 (local to host someguy123)
UUID : e49ab53b:c66321f0:9a4e272e:09dc25b1
Events : 93
Number Major Minor RaidDevice State
0 253 9 0 active sync set-A /dev/dm-9
1 253 10 1 active sync set-B /dev/dm-10
2 253 11 2 active sync set-A /dev/dm-11
3 253 12 3 active sync set-B /dev/dm-12
5 253 14 4 active sync set-A /dev/dm-14
4 253 13 5 active sync set-B /dev/dm-13
Solution 2:
There are 2 cases:
- you add new drives The easiest and safest way is to create a new array on the new drives , create a physical volume on the new array and here you are. No loss of performance here. As a bonus you can create a new volume group in order to put your data in one array or the other.
- You replace existing drives with bigger ones Replace them one by one, each time create 2 partitions on the disk. You add the first (for example sdX1 to the existing array ( it should recover automatically ) and you can then create a new array on all the second partions ( the sdX2 ). Depending on your usage there might be a performance hit for some operations ( basically if you copy data between both arrays ).
In both cases you won't lose data and if your hardware allows hotplug you will not have downtime.
By the way even if mdadm allowed a dynamic resize of the array I would not take the chance with production data.