3Ware 9650SE RAID-6, two degraded drives, one ECC, rebuild stuck

Solution 1:

I managed to get the RAID to rebuild by issuing the following command in tw_cli without pulling any drives or rebooting the system:

/c2/u1 set ignoreECC=on

The rebuild didn't proceed immediately, but at 2 AM the morning after I made this change, the rebuild started and about 6 hours later, it was complete. The drive with ECC errors had 24 bad sectors that have now been overwritten and reallocated by the drive (according to the SMART data). The filesystem seems intact, but I won't be surprised if I hit errors when I get to whatever data was on those sectors.

In any case, I'm much better off that I was before, and will likely be able to recover the majority of the data. Once I've gotten what I can I'll pop out the drive that's failing and have it rebuild onto a hot spare.