Question marks showing in ls of directory. IO errors too

Has anyone seen this before? I've got a raid 5 mounted on my server and for whatever reason it started showing this:

jason@box2:/mnt/raid1/cra$ ls -alh
ls: cannot access e6eacc985fea729b2d5bc74078632738: Input/output error
ls: cannot access 257ad35ee0b12a714530c30dccf9210f: Input/output error
total 0
drwxr-xr-x 5 root root 123 2009-08-19 16:33 .
drwxr-xr-x 3 root root  16 2009-08-14 17:15 ..
?????????? ? ?    ?      ?                ? 257ad35ee0b12a714530c30dccf9210f
drwxr-xr-x 3 root root  57 2009-08-19 16:58 9c89a78e93ae6738e01136db9153361b
?????????? ? ?    ?      ?                ? e6eacc985fea729b2d5bc74078632738

The md5 strings are actual directory names and not part of the error. The question marks are odd, and any directory with a question mark throws an io error when you attempt to use/delete/etc it.

I was unable to umount the drive due to "busy". Rebooting the server "fixed" it but it was throwing some raid errors on shutdown. I have configured two raid 5 arrays and both started doing this on random files. Both are using the following config:

mkfs.xfs -l size=128m -d agcount=32
mount -t xfs -o noatime,logbufs=8

Nothing too fancy, but part of an optimized config for this box. We're not partitioning the drives and that was suggested as a possible issue. Could this be the culprit?

Solution 1:

I had a similar problem because my directory had read (r) but not execute (x) rights. My directory listing showed:

myname@srv:/home$ ls -l service/mail/
ls: cannot access service/mail/001_SERVICE INBOX: Permission denied
total 0
-????????? ? ? ? ?                ? 001_SERVICE INBOX
d????????? ? ? ? ?                ? 01_CURRENT SERVICE

The mail directory had the r bit set, but not the x that you need for listing or search and access. Doing sudo chmod -R g+x mail solved this problem.

Solution 2:

The answers mentioning the read, but not execute or stat() are correct. But there is a common cause of this (other than corruption) that's bitten me a few times and would match your question with the IO errors nicely. If you improperly mount a filesystem, the mount point of that filesystem may show up with question marks. If you're seeing these where you've just tried mounting a new filesystem, try the following before worrying about corruption and fsck.

$ sudo umount /mnt/raid1/cra/257ad35ee0b12a714530c30dccf9210f
$ ls -alh /mnt/raid1/cra

You should see the 257ad35ee0b12a714530c30dccf9210f folder with permissions and attributes rather than question marks. If so, search for other options for your mount command or /etc/fstab file. If not, perhaps it is time to read the other answers, backup what you can, and run a fsck.

Solution 3:

The question marks in the ls output just indicate that it could not stat() the directory entry. You can also see those if you ls a directory for which you have r(ead) but not x (search) permission. However in that case it would not report I/O error.

In your case it looks like there is a disk error or possibly filesystem corruption. /var/log/messages or dmesg is likely to reveal further details.