After filesystem reached 100% storage capacity now set to read-only, how to reset to read-write mode?

Solution 1:

It is filesystem corruption that caused this switch to read only mode, rather that its overflow, exactly following the mount option errors=remount-ro.

Backup important data and configuration and download them somewhere. Prepare a recovery plan for the case if something important to boot was broken. If it's possible, move important services away to another machine. There'll be some downtime.

I noticed this system doesn't reboot very often (only 7 mounts since 2017, last reboot was in 2019). So I suggest setting up maximal mount count to 1, so it'll be checked each boot:

tune2fs -c 1 /dev/md3

Then reboot. Init script should check and fix the filesystem during boot. However, the corruptions might be so severe so it may require manual interaction, so assure someone is near the server and ready to help you. And, if something important was touched by this corruption, you may have strange problems.

In the worst case, you'll have to reinstall the system. But don't forget setting maximal mount count to 1 again.

Why was the file system corrupted? It just happens. Blocks are stored in memory and the corruption might have taken place there, due to say cosmic rays. Very rare occasion, it happens sometimes. Then, disks are also not ideal and cant detect all errors; there is nonzero bit error rates (look up actual value in your device datasheet), so there is very low chance for data to be read broken, but it is still possible. If this happens on metadata block, the problem may accumulate (file system driver directed by wrong information might do some incorrect assumption and broke the file system further), this is why it is important to check it from time to time.