USB storage media - data integrity issues on linux
I've always been somewhat paranoid about verifying data backed up to removable media, so after copying stuff to a USB flash drive or portable HDD, I invariably unmount the drive, remount it and diff -q the stored files with the originals.
Years ago I discovered that (at least with the equipment I've got), I was seeing bit errors at something on the order of 1bit/GByte. Somehow (I forget the details) I discovered that the cure is, before writing any data, to do
echo 64 > /sys/block/sda/device/max_sectors
(assuming the media appears as sda of course).
As long as I remember to do that, I've never had any problems.
(I belive the default max_sectors value is 128).
My questions are:
Is this just me ? I've seen the issue with a variety of flash drives, portable HDDs, motherboards and laptops (but never done an exhaustive test of all combinations to see if I have any which are actually reliable). The media which has been used with windows, and the machines which dual-boot windows, seem to have no similar problems there so it does appear to be linux specific.
What actually causes the issue ? Is non-standards-compliant media, chipsets, cables ?
Is there anything I can configure on my systems (Debian Lenny) which will automatically set the
max_sectors? (Some HAL scriptage or sysctl tweak ? A more global /sys parameter ?). Presumably the default 128 is in the kernel somewhere, but a custom kernel seems a bit drastic.
Thanks for any advice
First of all when you get a new device I can recommend writing data to it and verify the data afterwards using md5sum/sha1sum/... Especially cheap USB devices tend to be broken. :( It's not that unusual that USB pens work fine on the first few (hundred) MBs but tend to dataloss on the last MBs. Sadly many users aren't aware of that and notice problems too late: when the USB pen is getting full.
The problem you're speaking of is usually located at the USB chipset (though sometimes it's visible only when using special combinations of hardware). If it works with Windows but fails with Linux using the same hardware this sounds like there's a workaround in the Windows driver that does not exist in the Linux kernel (yet). And whereas Linux uses max_sectors=240 by default (being 120kB) it seems to be 64kB transfers (max_sectors=128) for Windows, according to http://www.linux-usb.org/FAQ.html#i5 (where some problems with the adapters made by Genesys Logic are mentioned as well).
To automatically set the max_sectors: use udev for this task. It allows you to configure max_sectors for just the devices you want to adjust it for.
You can retrieve the necesssary information about your usb device running:
# udevadm info -a -p /sys/class/block/sda/
or for older udev versions:
# udevinfo -a -p /sys/class/block/sda/
Then grab the attributes you'd like to use for your rules and create a new file like 42-fix-max_sectors.rules and place it under /lib/udev/rules.d (when using recent udev versions) or /etc/udev/rules.d (for older udev versions). To get an idea how this configuration file could look like:
SUBSYSTEM=="block", ATTRS{model}=="Voyager Mini ", RUN+="/bin/sh -c '/bin/echo 64 > /sys/block/%k/device/max_sectors'"
Make sure to reload udev after writing your rules file (through running /etc/init.d/udev reload). For testing your configuration file I can recommend using:
# udevadm test /sys/class/block/sda/
PS: I prefer to replace hardware if I notice any problems because usually it's not worth the effort to work on any workarounds (and I'm sure you're aware of murphy who will catch you anyway as soon as you think you've got it working ;)).
Depending on the number of files or size of files, I have seen similar things in the past.
During a backup/copy of over 4 million .PDF files we found that the MD5 hashes on some files were NOT the same!
What worked for us was rsync.
One thing I would suggest you try is to rsync the files over and see if you still experience the data loss.
Hope this helps.