Huge amounts of "multiply-claimed blocks" during fsck
Solution 1:
How long it takes would depend on disk subsystem performance, the damage being repaired, etc.
It sounds like there's some decent filesystem corruption. How big is the actual filesystem? You said it's a 100 GB file and later that it's a VM image? Is this a VM server? Or are you talking about virtualbox?
Personally if it took over a day and the damage was definitely to one file, I'd restore the file from backup and if there were any indications of continuing issues, reformat and restore from backup, assuming the drive isn't coincidentally failing. I have trust issues with filesystems that start to go bad. If the drive itself isn't failing, the filesystem may have a pervasive problem until it's started fresh.
But that's me.
Solution 2:
It seems as thought the cause for the large amount of time and also the mystery of the multiply-claimed blocks being shared by zero files was the result of a degraded RAID array.
As soon as I removed the faulty drive, the fsck went much faster. There were still a few multiply-claimed blocks but they were fixed very quickly.
I have experienced degraded RAID arrays in Ubuntu before and usually there is a warning just after the grub phase, but this did not happen in this case.
Solution 3:
This happens to me on RAID array of 6 disks, 4.5TB ext4 filesystem. Linux 3.3.7-1-ARCH #1 SMP PREEMPT i686
I use rsync to sync over entire servers onto the ext4 and these are the files I mostly get these multiply-claimed-blocks and duplicate inode messages on.
EXT4 Mount Options
A couple things I did that seemed to help was to make sure that the ext4 was being mounted with barrier and data=ordered support.
/dev/md5 /md5 ext4 defaults,noatime,nouser_xattr,stripe=1536,data=ordered 0 2
RAID Bitmap
The other step I took was to enable bitmap on the RAID.
mdadm --grow /dev/md5 --bitmap=internal
or
mdadm --grow /dev/md5 --bitmap=/external/md5.bitmap
Having both the raid bitmap and ext4 journal located on an external device seems to work the best.
USB Autosuspend
In the past I had experienced this issue when my drives would go into autosuspend mode. Writing to them (or attempting to) while they were trying to wake up from a suspended state seemed to cause these issues big time. What I did was disable autosuspend compeletely on usb devices with:
usbcore.autosuspend=-1
EXT4 Data Modes
From: http://kernel.org/doc/Documentation/filesystems/ext4.txt
There are 3 different data modes:
writeback mode In data=writeback mode, ext4 does not journal data at all. This mode provides a similar level of journaling as that of XFS, JFS, and ReiserFS in its default mode - metadata journaling. A crash+recovery can cause incorrect data to appear in files which were written shortly before the crash. This mode will typically provide the best ext4 performance.
ordered mode In data=ordered mode, ext4 only officially journals metadata, but it logically groups metadata information related to data changes with the data blocks into a single unit called a transaction. When it's time to write the new metadata out to disk, the associated data blocks are written first. In general, this mode performs slightly slower than writeback but significantly faster than journal > mode.
journal mode data=journal mode provides full data and metadata journaling. All new data is written to the journal first, and then to its final location. In the event of a crash, the journal can be replayed, bringing both data and metadata into a consistent state. This mode is the slowest except when data needs to be read from and written to disk at the same time where it outperforms all others modes. Currently ext4 does not have delayed allocation support if this data journalling mode is selected.
Fix with Debugfs
This has great examples to fix: http://www.redhat.com/archives/ext3-users/2009-February/msg00021.html
Solution 4:
I think, I had a similar problam. I have 2 HDD in a RAID0 array. Once, I did a fsck manually, after I unmounted the device. To my pain, I didn't realize, that a VM was still running and accessing the device during I fsck'ed. The result was lots of multiply claimed blocks and because I rebooted my server during the progress, I think it broke the superblock. So I couldn't even mount the RAID anymore.
I fixed the issue by restoring the superblock, once again by running fsck and repairing all issues which didn't have anything to do with "multiply claimed blocks". This took me some time and I needed to attend the process, to tell fsck not to repair the "multiply claimed blocks".
After that, the superblock was fixed and I could mount the device once again. Now, I ran fsck several times and checked, which files were affected by "multiply claimed blocks", stopped the process by hitting ctrl^c and simply copied the affected files and deleted the original once.
Sounds unconventional, but it fixed my problems in a quick way and my HDDs seem to be clean (according to e2fsck).
If there was a better/faster way to fix these to problems, I am glad to hear about them.