Why is this SSD returning inconsistent data, why doesn't the backup image file match the checksum?
This is about the SSD in a notebook. It appears the SSD is already going bad, possibly even corrupting data. It appears to return different data everytime it's accessed while not in use (see below for details). Which tools can be used to confirm this suspicion?
When an HDD slowly begins to die, there's usually a clear indication in the SMART output, a graphical tool like gsmart control would highlight a certain value in red and a service like smartd might already have generated a warning. At that point, the user might still have some time to create a backup before the drive starts to corrupt data. Of course, if the drive has already begun to corrupt data, some files in that backup could be damaged.
It seems there's no clear warning in the SMART output for this SSD, no kernel errors have been logged to dmesg etc (on the other hand, ecryptfs has logged errors). In other words, it was only by chance discovered that this SSD might already be so bad that it's corrupting data even when it's not in use.
After making a backup (1:1 dd image) of this SSD (sda), I discovered that the checksum of the image file doesn't match the checksum of the SSD. Of course, this was in a live system, so the SSD was not mounted, which means its contents could not have changed during the backup process.
This is what I did to make the backup copy. "BUTTER" is where I mounted an external drive, which is formatted with BTRFS so that I would be able to find out about data errors in case the backup drive is also bad (unlike most other filesystems, BTRFS has checksums).
[root@localhost mnt]# time dd if=/dev/sda of=BUTTER/SSD.dd.img bs=400M && echo OK
610+1 records in
610+1 records out
256060514304 bytes (256 GB, 238 GiB) copied, 5188.27 s, 49.4 MB/s
real 86m28.726s
user 0m0.008s
sys 7m3.597s
OK
I created an MD5 checksum of the image file and another one of the SSD and they didn't match. After repeating this procedure, I realized that the MD5 checksum of the SSD is different every single time.
[root@localhost mnt]# time dd if=/dev/sda bs=400M | md5sum >/tmp/MD5-again
610+1 records in
610+1 records out
256060514304 bytes (256 GB, 238 GiB) copied, 1059.87 s, 242 MB/s
real 17m39.904s
user 8m21.708s
sys 3m58.466s
[root@localhost mnt]# cat /tmp/MD5-again
24e71715359158f3ab38e748af93718c -
[root@localhost mnt]# time dd if=/dev/sda bs=400M | md5sum >>/tmp/MD5-again
610+1 records in
610+1 records out
256060514304 bytes (256 GB, 238 GiB) copied, 1073.7 s, 238 MB/s
real 17m53.735s
user 8m28.494s
sys 4m23.993s
[root@localhost mnt]# cat /tmp/MD5-again
24e71715359158f3ab38e748af93718c -
569d517626c1b7394acca0c4020c99bc -
Again, the SSD was never mounted at any point during that process.
# mount | grep -c sda
0
I ran a SMART test, which did not find anything. No SMART error is logged.
SMART attributes:
Device Model: SanDisk SD8TN8U256G1001
[root@localhost mnt]# smartctl -A /dev/sda
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.16.3-301.fc28.x86_64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 4
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0032 100 100 --- Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 --- Old_age Always - 3173
12 Power_Cycle_Count 0x0032 100 100 --- Old_age Always - 1117
170 Unknown_Attribute 0x0032 100 100 --- Old_age Always - 0
171 Unknown_Attribute 0x0032 100 100 --- Old_age Always - 0
172 Unknown_Attribute 0x0032 100 100 --- Old_age Always - 0
173 Unknown_Attribute 0x0032 100 100 --- Old_age Always - 37
174 Unknown_Attribute 0x0032 100 100 --- Old_age Always - 41
178 Used_Rsvd_Blk_Cnt_Chip 0x0032 100 100 --- Old_age Always - 0
180 Unused_Rsvd_Blk_Cnt_Tot 0x0033 100 100 010 Pre-fail Always - 100
184 End-to-End_Error 0x0033 100 100 097 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 --- Old_age Always - 0
194 Temperature_Celsius 0x0022 056 062 --- Old_age Always - 44 (Min/Max 13/62)
199 UDMA_CRC_Error_Count 0x0032 100 100 --- Old_age Always - 0
233 Media_Wearout_Indicator 0x0033 093 100 001 Pre-fail Always - 15484248
234 Unknown_Attribute 0x0032 100 100 --- Old_age Always - 11127
241 Total_LBAs_Written 0x0030 253 253 --- Old_age Offline - 3192
242 Total_LBAs_Read 0x0030 253 253 --- Old_age Offline - 66461
249 Unknown_Attribute 0x0032 100 100 --- Old_age Always - 9346
What's happening?
Solution 1:
Right after posting this question, I found my mistake. I used Fedora XFCE as live system, which has automatically enabled the swap partition that just happens to be on the SSD in question. And of course, while the live system was actively using the swap partition on the SSD, it was thereby changing the contents of the SSD.
[root@localhost mnt]# swapon --show
NAME TYPE SIZE USED PRIO
/dev/sda3 partition 8G 103.3M -2
That's a bit awkward now that I've already posted the question. But I'll leave it there, hoping it will be useful for someone else who might be doing the same mistake.
All I had to do was disable the swap partition that was previously automatically mounted:
[root@localhost mnt]# swapoff -a
I'd like to point out that the swap partition was mounted automatically when I booted the live system. I didn't want that swap partition to be mounted. I wonder what happens if there was a hibernate image on that swap partition.
After disabling the unwanted swap partition, everything worked as expected. Using the commands shown above, the image file's checksum now matches the checksum of the SSD. In other words, this SSD isn't bad.