input/output error at clean ext3 partition - how to check what wrong with data block

Solution 1:

So, to figure this out I did the following.

Take your block number, multiply by four and add one

(130856866 * 4) + 1 = 523427465

This represents the sector reported as producing an I/O error. The block size being 2k, sectors being 512 bytes. The additional one extra accounts for the starting sector offset for the partition.

To correlate with SMART, we'll need to convert the value we now have into hexadecimal.

$ printf "0x%x\n" 523427465
0x1f32de89

Now, when you correlate that with what SMART shows, a line comes up suspiciously close..

20 34463:43  810000001f32decd  [3,11,0]   Require Write or Reassign Blocks command

How far away?

$ bc -l
bc 1.06.95
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 
obase=16
ibase=16
1F32DECD-1F32DE89
44

That works out as being only between 34816 and 32768 bytes away but we cant say which sector is damaged out of the four that comprise the block.

If I had to hazard a guess, I'd say that theres probably a whole slew of blocks around the same address which will report I/O errors (assuming the raid striping is say 32k in size or whatever).

Additionally a read may not pick up the problem if the RAID is fetching the block chunk from a different disk. A write must propagate to all the disks in a RAID1 setup anyhow so this could make writes fail but reads succeed. Additionally, if we assume that the chunk size of the RAID card is 32k we can also assume that the damaged block plus the one reported by SMART are both damaged by whatever happened on that platter. Its just the SMART test read from the good disk for the first 32k and the bad disk for the next 32k.

Modern hard disks keep 'reserve sectors' to replace damaged sectors like this with a new sector location. Seeing as you are now getting this, and the that Reassign by disk failed message from smart I'd say a disk has ran out.

In terms of doing something about it; thats a bit more trickier. The LBA addressing is an abstraction against the real disk underneath. You'd need to identify which disk it is that is causing this issue, fail it in the RAID array and replace it.

In any case, you have a bad disk and you should look to replace it ASAP.

Solution 2:

That's a lot to process... But a few things jump out at me.

Your kernel version is: 2.6.18-164.15.1.el5 - That places your kernel revision at the EL5.4 level, or circa March 2010.

I had persistent ext3 filesystem stability and corruption issues in EL5. Things weren't fully fixed until mid-2012. In my worst situation, I was working with a cloud infrastructure firm that never updated kernels from their base release. So I began to see these problems at scale across thousands of EL5 servers.

Is there any chance you can update your OS/kernel/e2fsprogs, fsck and try again?

In addition, if the kernel is circa 2010, your system's BIOS and Smart Array P410 firmware is probably very outdated. What model server is this?

Edit:

Your cciss CHECK_CONDITION errors are the giveaway. No need to even deal with SMART at this point. Run the HP Array Diagnostic Utility and it will distill the error information into a report. Either way, I really hope this isn't a RAID5 array.

Can you post the output of hpacucli ctrl all show config detail ?

Solution 3:

The actually failed blocks are readable from the kernel log, which you can read somewhere below /var/log (probably /var/log/kernel.log), or from the output of the dmesg command.

Beware: what you need is not the disk sector number, but the partition- and filesystem-specific block number. Kernels since around 2.4.x are saying both of them in the dmesg.

Giving an -L flag to e2fsck can give this list of blocks to the bad blocks list of the filesystem. Thus the correct steps are the following:

First, check the list of the bad blocks from a dmesg.

Second, put them into a simple text file, so

cat >badblockfile.txt
34252345
3452345
23452345

(ctrl/d)

e2fsck -f -y -C0 /dev/diskname -L badblockfile.txt

If you can't interpret dmesg, put the relevant parts here as comments or as extension of your question.

EXTENSION

Your filesystem has 2k-blocks, and starts on the first sector of your hard disk (which has 512byte-sectors). Thus the formula between the filesystem-blocks (which could be given to e2fsck), and the disk-block (which are in the dmesg output) if very simple:

filesystem_block=(serctor_no-1)/4

If you don't have filesystem-level blocks in your messages, you could use this formula as well.

ALTERNATE TIP

There is also an additional tip: e2fsck has a flag -c. This calls the tool badblocks before a check, and marks the newly found bad blocks as bad. As I experienced, it doesn't really okay, in most cases it can't find all of the bad blocks. In your place I runned this for the weekend (or at least for the night) in an infinite loop:

while true; do e2fsck -f -y -C0 -c /dev/sdf;done