mdadm RAID 5 Failed with 2 Drives while rebuilding
There is no backup.. This is the problem.
Storing important data (on ANY system, no matter how reliable) without a backup is indeed the problem!
Having no backup, and having experienced a failure mode for RAID 5 for which there is no proper recovery path, you are now what our British friends would refer to as "Right Royally Rogered" (actually they would probably use more colorful language).
You're down to two options at this point:
- Cry.
- Contact a data recovery company, explain what happened, and deliver to them a princely ransom to attempt a dark magic ritual to extract your data from the clutches of your failed disks.
(1 and 2 are not mutually exclusive - In fact when you see the price for (2) you will probably do (1)...)
You can consider this a learning experience, and an expensive object lesson in the importance of regular backups and restore testing...
I just wanted to update everyone on the solution.
End result i got all my data back except for 2 files and here is what i did
- Installed the good drive and the drive that failed during the rebuild
- Forced mdadm to create the array with 1 drive missing and set the flag so that all disks are clean
- used MC (midnight commander) to start copying files folder by folder. Once mdadm detected an error on the array it spit out some errors, mdadm would removed the drive from the array, thus making the array unable. MC would then give me a read error ( Perfect because i didn't want it to keep coping corrupt data ). Take note of the file it got hung up on
- Unmounted the array, stop mdadm, and then restart it again with 2 drives, mark them as clean
- Skipped OVER the file it got stuck on.
Anyways i was able to recover all data except for 2 files. I hope this helps anyone that gets into this situation. I have also went back to a RAID-1. At least with RAID-1 if 1 drive has bad sectors i can still recover the data without having to use mdadm.
PS - I have learned my lesson and also added in backups to another drive.
Thanks to everyone.
Never use raid 5 on slow disks, and never use it on software raid. Also never use it unless you have a decent raid card with its own cache and pre-failure analysis.