Buffer I/O Error on md-device - can't identify failed drive
Syncing my postgres master to the slave server resulted in write I/O errors on the slave (journalctl):
Aug 18 03:09:23 db01a kernel: EXT4-fs warning (device dm-3):
**ext4_end_bio:330: I/O error -5 writing to inode 86772956 (offset 905969664 size 8388608 starting block 368694016)**
Aug 18 03:09:23 db01a kernel: buffer_io_error: 326 callbacks suppressed
....
Reading the affected file of course also doesn't work:
cat base/96628250/96737718 >> /dev/null
cat: 96737718: Input/output error
Shouldn't the linux kernel (ubuntu 16.04 4.4.0-87-generic) kick the affected drive from the array automatically?
As it is a Raid6 (LVM and ext4 on top) I already tried to overwrite every SSD a few times with badblocks to provoke the error (removed one disk after another from the raid for that), unfortunately with no success.
smartctl says one disk had errors before (the others are clean):
smartctl -a /dev/sda
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 099 099 010 Pre-fail Always - 2
179 Used_Rsvd_Blk_Cnt_Tot 0x0013 099 099 010 Pre-fail Always - 2
183 Runtime_Bad_Block 0x0013 099 099 010 Pre-fail Always - 2
187 Uncorrectable_Error_Cnt 0x0032 099 099 000 Old_age Always - 3
195 ECC_Error_Rate 0x001a 199 199 000 Old_age Always - 3
But rewriting the whole disk with badblocks -wsv worked without error.
As it is a pretty important server for me, I replaced the whole server with a different model, but the error persisted. Am I correct in thinking that it's probably a disk issue?
Is there any way to know which disk is affected, maybe by calculating?
EDIT: For clarification: What I'm not getting is how the initial sync of 1.5 TB data from the master to the slave can result in two unrecoverable I/O errors, but manually running destructive read-write tests on every involved SSD completes without any error.
EDIT2: Output of lsblk (identical for sda-sdf); pvs; vgs; lvs;
lsblk:
sda1 8:16 0 953.9G 0 disk
├─sda1 8:17 0 4.7G 0 part
│ └─md0 9:0 0 4.7G 0 raid1
└─sda5 8:21 0 949.2G 0 part
└─md1 9:1 0 2.8T 0 raid6
├─vgdb01a-lvroot 252:0 0 18.6G 0 lvm /
├─vgdb01a-lvvar 252:1 0 28G 0 lvm /var
├─vgdb01a-lvtmp 252:2 0 4.7G 0 lvm /tmp
└─vgdb01a-lvpostgres 252:3 0 2.6T 0 lvm /postgres
pvs:
PV VG Fmt Attr PSize PFree
/dev/md1 vgdb01a lvm2 a-- 2.78t 133.64g
vgs:
VG #PV #LV #SN Attr VSize VFree
vgdb01a 1 4 0 wz--n- 2.78t 133.64g
lvs:
lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lvpostgres vgdb01a -wi-ao---- 2.60t
lvroot vgdb01a -wi-ao---- 18.62g
lvtmp vgdb01a -wi-ao---- 4.66g
lvvar vgdb01a -wi-ao---- 27.94g
Update 2017-8-22
echo check > /sys/block/md1/md/sync_action
[Mon Aug 21 16:10:22 2017] md: data-check of RAID array md1
[Mon Aug 21 16:10:22 2017] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[Mon Aug 21 16:10:22 2017] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
[Mon Aug 21 16:10:22 2017] md: using 128k window, over a total of 995189760k.
[Mon Aug 21 18:58:18 2017] md: md1: data-check done.
echo repair > /sys/block/md1/md/sync_action [Tue Aug 22 12:54:11 2017] md: requested-resync of RAID array md1
[Tue Aug 22 12:54:11 2017] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[Tue Aug 22 12:54:11 2017] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for requested-resync.
[Tue Aug 22 12:54:11 2017] md: using 128k window, over a total of 995189760k.
[2160302.241701] md: md1: requested-resync done.
e2fsck -y -f /dev/mapper/vgdb01a-lvpostgres
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/vgdb01a-lvpostgres: 693517/174489600 files (1.6% non-contiguous), 608333768/697932800 blocks
Update 2017-8-22 2 Output of lsscsi and all disk smartctl on pastebin: https://pastebin.com/VUxKEKiF
UPDATE-8/22
If you want to solve this problem quickly just replace the two drives that have the worst smartctl stats and reassess. Once you're out of reserved blocks your drive is EOL. Seeing that we buy these all at once they tend to fail around the same time. So it doesn't matter which one is the bad block is pinned to. Once you replace the worst two offenders (that means replace one and resync and repeat), you'll have increased the overall health of the array, probably replaced the complaining disk, and dramatically reduced the risk of a double fault where you lose everything.
At the end of the day, your data is worth more than a few hundred bucks.
ENDUPDATE-8/22
UPDATE-8/21
Toni Yes, your original post has room for improvement. Given those facts this is the conclusion I arrived to. It also wasn't clear until now that you already suffered data corruption.
It would be helpful if you included the headers with the smartctl output. This is easier on scsi, sg_reassign will tell you how many free blocks you have left to reassign, once that goes to zero, you're done. Seeing that smartctl is reporting "prefail" in several categories it sounds like you're done soon too.
Soon you'll experience hard media errors and MD will kick the drive. fsck would be a good idea in the meanwhile. When drives fail a write they reassign the destination from the free block pool, when you run out, it becomes a unrecoverable media error.
Also enable "disk scrubber" on MD and run it on cron weekly, it will read and rewrite every sector and head this off before it becomes a real problem. See Documentation/md.txt in the kernel.
[disk scrubber example] https://www.ogre.com/node/384
You still have to run smartmon all the drives (once a day, off hours), parse the output, and create alarms to head off this very problem.
Folks, this is what hardware raids do for you. The irony is, we have all the tools to provide a better MD experience, but no one puts it together into a integrated solution.
You're pretty much at the tail end of silent data corruption. A fsck might help you, but really the best thing to do is refer to your backups (you kept backups right? RAIDs are not backups) and prepare for this RAID to start sinking.
Then you'll find the bad disk.
Sorry.
ENDUPDATE-8/21
For starters, did you read the man page for badblocks for the options you used?
-w Use write-mode test. With this option, badblocks scans for bad blocks by writing
some patterns (0xaa, 0x55, 0xff, 0x00) on every block of the device, reading every
block and comparing the contents. This option may not be combined with the -n
option, as they are mutually exclusive.
So your data is gone, -n was the nondestructive version. Maybe what you really did was pull a disk from the array, run badblocks on it, and then reinserted it? Please clarify.
That you don't know which disk is failed to begin with tells me that it is not an MD raid array. So whatever non-existent lvm "raid" tools exist to help you recover from this simple failure, that's what you need to figure out.
I would say that the majority of users go with an MD raid solution. The remainder get distracted by "what's this thing?" or "oh, this is LVM, it's what I'm supposed to do, right?" and later end up where you are now. I raid implementation with terrible management tools which actually created more risk than you attempted to mitigate by building a RAID 6 to begin with.
It's not your fault, you didn't know. Frankly, they should disable the thing for exactly this reason.
Concerning repairing bad blocks. You can do this by taking the machine offline and booting to a live usb drive and performing one of the following repair procedures.
https://sites.google.com/site/itmyshare/storage/storage-disk/bad-blocks-how-to
http://linuxtroops.blogspot.com/2013/07/how-to-find-bad-block-on-linux-harddisk.html
As to where this sector is in your array. Well, you would have to account for the parity rotation, which is a PITA. I would suggest that you simply verify each drive until you find the problem.
You can help prevent this in the future by enabling "disk scrubbing" in MD which reads and rewrites each sector in a maintenance window to discover exactly these sort of problems and potentially repair them.
I hope this helps.