Is my SSD drive failing?
I'm using a fairly new SSD (bought 5 months ago), and I recently experienced a read failure. Here is what the kernel reported:
Jul 1 01:02:44 tryphon kernel: [1554891.463590] ata2.00: exception Emask 0x0 SAct 0x1000000 SErr 0x0 action 0x0
Jul 1 01:02:44 tryphon kernel: [1554891.478096] ata2.00: irq_stat 0x40000008
Jul 1 01:02:44 tryphon kernel: [1554891.486388] ata2.00: failed command: READ FPDMA QUEUED
Jul 1 01:02:44 tryphon kernel: [1554891.497141] ata2.00: cmd 60/80:c0:00:18:2c/05:00:02:00:00/40 tag 24 ncq dma 720896 in
Jul 1 01:02:44 tryphon kernel: [1554891.497141] res 41/40:80:00:18:2c/00:05:02:00:00/40 Emask 0x409 (media error) <F>
Jul 1 01:02:44 tryphon kernel: [1554891.530266] ata2.00: status: { DRDY ERR }
Jul 1 01:02:44 tryphon kernel: [1554891.538714] ata2.00: error: { UNC }
Jul 1 01:02:44 tryphon kernel: [1554891.586520] ata2.00: configured for UDMA/133
Jul 1 01:02:44 tryphon kernel: [1554891.586576] sd 1:0:0:0: [sdb] tag#24 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 1 01:02:44 tryphon kernel: [1554891.586581] sd 1:0:0:0: [sdb] tag#24 Sense Key : Medium Error [current]
Jul 1 01:02:44 tryphon kernel: [1554891.586585] sd 1:0:0:0: [sdb] tag#24 Add. Sense: Unrecovered read error - auto reallocate failed
Jul 1 01:02:44 tryphon kernel: [1554891.586590] sd 1:0:0:0: [sdb] tag#24 CDB: Read(10) 28 00 02 2c 18 00 00 05 80 00
Jul 1 01:02:44 tryphon kernel: [1554891.586593] blk_update_request: I/O error, dev sdb, sector 36444160
A few minutes later, the smartd daemon sent an email to root:
Subject: SMART error (OfflineUncorrectableSector) detected on host: tryphon
This message was generated by the smartd daemon running on:
host name: tryphon
DNS domain: home
The following warning/error was logged by the smartd daemon:
Device: /dev/sdb [SAT], 1 Offline uncorrectable sectors
Device info:
LDLC, S/N:07072217A1171, FW:Q0601B, 120 GB
For details see host's SYSLOG.
You can also use the smartctl utility for further investigation.
Another message will be sent in 24 hours if the problem persists.
Since it is used in a RAID 1 configuration with an HDD, this is not dramatically bad, but it still sounds bad.
Anyway, I wanted to check the readability of my SSD with dd if=/dev/sdb of=/dev/null and experienced more read failures. Here are the kernel logs for the next read errors:
Jul 3 19:28:24 tryphon kernel: [ 4398.319485] ata2.00: exception Emask 0x0 SAct 0x180 SErr 0x0 action 0x0
Jul 3 19:28:24 tryphon kernel: [ 4398.332971] ata2.00: irq_stat 0x40000008
Jul 3 19:28:24 tryphon kernel: [ 4398.340961] ata2.00: failed command: READ FPDMA QUEUED
Jul 3 19:28:24 tryphon kernel: [ 4398.351416] ata2.00: cmd 60/00:38:08:23:80/01:00:00:00:00/40 tag 7 ncq dma 131072 in
Jul 3 19:28:24 tryphon kernel: [ 4398.351416] res 41/40:00:08:23:80/00:01:00:00:00/40 Emask 0x409 (media error) <F>
Jul 3 19:28:24 tryphon kernel: [ 4398.383720] ata2.00: status: { DRDY ERR }
Jul 3 19:28:24 tryphon kernel: [ 4398.391853] ata2.00: error: { UNC }
Jul 3 19:28:24 tryphon kernel: [ 4398.439236] ata2.00: configured for UDMA/133
Jul 3 19:28:24 tryphon kernel: [ 4398.439258] sd 1:0:0:0: [sdb] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 3 19:28:24 tryphon kernel: [ 4398.439261] sd 1:0:0:0: [sdb] tag#7 Sense Key : Medium Error [current]
Jul 3 19:28:24 tryphon kernel: [ 4398.439264] sd 1:0:0:0: [sdb] tag#7 Add. Sense: Unrecovered read error - auto reallocate failed
Jul 3 19:28:24 tryphon kernel: [ 4398.439268] sd 1:0:0:0: [sdb] tag#7 CDB: Read(10) 28 00 00 80 23 08 00 01 00 00
Jul 3 19:28:24 tryphon kernel: [ 4398.439270] blk_update_request: I/O error, dev sdb, sector 8397576
Jul 3 19:28:24 tryphon kernel: [ 4398.451841] ata2: EH complete
Jul 3 19:28:24 tryphon kernel: [ 4398.831494] ata2.00: exception Emask 0x0 SAct 0x1000 SErr 0x0 action 0x0
Jul 3 19:28:24 tryphon kernel: [ 4398.845137] ata2.00: irq_stat 0x40000008
Jul 3 19:28:24 tryphon kernel: [ 4398.853069] ata2.00: failed command: READ FPDMA QUEUED
Jul 3 19:28:24 tryphon kernel: [ 4398.863461] ata2.00: cmd 60/08:60:78:23:80/00:00:00:00:00/40 tag 12 ncq dma 4096 in
Jul 3 19:28:24 tryphon kernel: [ 4398.863461] res 41/40:08:78:23:80/00:00:00:00:00/40 Emask 0x409 (media error) <F>
Jul 3 19:28:24 tryphon kernel: [ 4398.895577] ata2.00: status: { DRDY ERR }
Jul 3 19:28:24 tryphon kernel: [ 4398.903692] ata2.00: error: { UNC }
Jul 3 19:28:24 tryphon kernel: [ 4398.951089] ata2.00: configured for UDMA/133
Jul 3 19:28:24 tryphon kernel: [ 4398.951111] sd 1:0:0:0: [sdb] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 3 19:28:24 tryphon kernel: [ 4398.951116] sd 1:0:0:0: [sdb] tag#12 Sense Key : Medium Error [current]
Jul 3 19:28:24 tryphon kernel: [ 4398.951119] sd 1:0:0:0: [sdb] tag#12 Add. Sense: Unrecovered read error - auto reallocate failed
Jul 3 19:28:24 tryphon kernel: [ 4398.951125] sd 1:0:0:0: [sdb] tag#12 CDB: Read(10) 28 00 00 80 23 78 00 00 08 00
Jul 3 19:28:24 tryphon kernel: [ 4398.951128] blk_update_request: I/O error, dev sdb, sector 8397688
Jul 3 19:28:24 tryphon kernel: [ 4398.963674] Buffer I/O error on dev sdb, logical block 1049711, async page read
Jul 3 19:28:24 tryphon kernel: [ 4398.978559] ata2: EH complete
Jul 3 19:28:24 tryphon kernel: [ 4399.051447] ata2.00: exception Emask 0x0 SAct 0x80000 SErr 0x0 action 0x0
Jul 3 19:28:24 tryphon kernel: [ 4399.065213] ata2.00: irq_stat 0x40000008
Jul 3 19:28:24 tryphon kernel: [ 4399.073159] ata2.00: failed command: READ FPDMA QUEUED
Jul 3 19:28:24 tryphon kernel: [ 4399.083555] ata2.00: cmd 60/08:98:78:23:80/00:00:00:00:00/40 tag 19 ncq dma 4096 in
Jul 3 19:28:24 tryphon kernel: [ 4399.083555] res 41/40:08:78:23:80/00:00:00:00:00/40 Emask 0x409 (media error) <F>
Jul 3 19:28:24 tryphon kernel: [ 4399.115655] ata2.00: status: { DRDY ERR }
Jul 3 19:28:24 tryphon kernel: [ 4399.123792] ata2.00: error: { UNC }
Jul 3 19:28:24 tryphon kernel: [ 4399.171197] ata2.00: configured for UDMA/133
Jul 3 19:28:24 tryphon kernel: [ 4399.171223] sd 1:0:0:0: [sdb] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 3 19:28:24 tryphon kernel: [ 4399.171228] sd 1:0:0:0: [sdb] tag#19 Sense Key : Medium Error [current]
Jul 3 19:28:24 tryphon kernel: [ 4399.171232] sd 1:0:0:0: [sdb] tag#19 Add. Sense: Unrecovered read error - auto reallocate failed
Jul 3 19:28:24 tryphon kernel: [ 4399.171237] sd 1:0:0:0: [sdb] tag#19 CDB: Read(10) 28 00 00 80 23 78 00 00 08 00
Jul 3 19:28:24 tryphon kernel: [ 4399.171240] blk_update_request: I/O error, dev sdb, sector 8397688
Jul 3 19:28:24 tryphon kernel: [ 4399.183787] Buffer I/O error on dev sdb, logical block 1049711, async page read
Jul 3 19:28:24 tryphon kernel: [ 4399.198659] ata2: EH complete
and:
Jul 5 12:28:06 tryphon kernel: [ 5.513650] ata2.00: exception Emask 0x0 SAct 0x4000 SErr 0x0 action 0x0
Jul 5 12:28:06 tryphon kernel: [ 5.527221] ata2.00: irq_stat 0x40000008
Jul 5 12:28:06 tryphon kernel: [ 5.535114] ata2.00: failed command: READ FPDMA QUEUED
Jul 5 12:28:06 tryphon kernel: [ 5.545448] ata2.00: cmd 60/08:70:98:af:32/00:00:00:00:00/40 tag 14 ncq dma 4096 in
Jul 5 12:28:06 tryphon kernel: [ 5.545448] res 41/40:08:98:af:32/00:00:00:00:00/40 Emask 0x409 (media error) <F>
Jul 5 12:28:06 tryphon kernel: [ 5.577373] ata2.00: status: { DRDY ERR }
Jul 5 12:28:06 tryphon kernel: [ 5.585423] ata2.00: error: { UNC }
Jul 5 12:28:06 tryphon kernel: [ 5.632752] ata2.00: configured for UDMA/133
Jul 5 12:28:06 tryphon kernel: [ 5.632778] sd 1:0:0:0: [sdb] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 5 12:28:06 tryphon kernel: [ 5.632784] sd 1:0:0:0: [sdb] tag#14 Sense Key : Medium Error [current]
Jul 5 12:28:06 tryphon kernel: [ 5.632790] sd 1:0:0:0: [sdb] tag#14 Add. Sense: Unrecovered read error - auto reallocate failed
Jul 5 12:28:06 tryphon kernel: [ 5.632797] sd 1:0:0:0: [sdb] tag#14 CDB: Read(10) 28 00 00 32 af 98 00 00 08 00
Jul 5 12:28:06 tryphon kernel: [ 5.632801] blk_update_request: I/O error, dev sdb, sector 3321752
Jul 5 12:28:06 tryphon kernel: [ 5.645296] md/raid1:md1: sdb3: rescheduling sector 173976
Jul 5 12:28:06 tryphon kernel: [ 5.656333] ata2: EH complete
Jul 5 12:28:06 tryphon kernel: [ 5.695829] md/raid1:md1: redirecting sector 173976 to other mirror: sda3
Then, I had a look at what the drive reports with the S.M.A.R.T. system. This is the result:
# smartctl -a /dev/sdb
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.9.0-0.bpo.6-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: LDLC
Serial Number: 07072217A1171
Firmware Version: Q0601B
User Capacity: 120,034,123,776 bytes [120 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 T13/2015-D revision 3
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Jul 11 00:43:45 2018 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x11) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities: (0x0002) Does not save SMART data before
entering power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 10) minutes.
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x0032 100 100 050 Old_age Always - 0
5 Reallocated_Sector_Ct 0x0032 100 100 050 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 050 Old_age Always - 3828
12 Power_Cycle_Count 0x0032 100 100 050 Old_age Always - 18
160 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 6
161 Unknown_Attribute 0x0033 100 100 050 Pre-fail Always - 99
163 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 60
164 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 3268
165 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 2
166 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 1
167 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 1
168 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 1500
169 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 100
175 Program_Fail_Count_Chip 0x0032 100 100 050 Old_age Always - 0
176 Erase_Fail_Count_Chip 0x0032 100 100 050 Old_age Always - 0
177 Wear_Leveling_Count 0x0032 100 100 050 Old_age Always - 18
178 Used_Rsvd_Blk_Cnt_Chip 0x0032 100 100 050 Old_age Always - 1
181 Program_Fail_Cnt_Total 0x0032 100 100 050 Old_age Always - 0
182 Erase_Fail_Count_Total 0x0032 100 100 050 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 050 Old_age Always - 7
194 Temperature_Celsius 0x0022 100 100 050 Old_age Always - 40
195 Hardware_ECC_Recovered 0x0032 100 100 050 Old_age Always - 11868576
196 Reallocated_Event_Count 0x0032 100 100 050 Old_age Always - 6
197 Current_Pending_Sector 0x0032 100 100 050 Old_age Always - 0
198 Offline_Uncorrectable 0x0032 100 100 050 Old_age Always - 6
199 UDMA_CRC_Error_Count 0x0032 100 100 050 Old_age Always - 0
232 Available_Reservd_Space 0x0032 100 100 050 Old_age Always - 99
241 Total_LBAs_Written 0x0030 100 100 050 Old_age Offline - 10975
242 Total_LBAs_Read 0x0030 100 100 050 Old_age Offline - 29075
245 Unknown_Attribute 0x0032 100 100 050 Old_age Always - 2904
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
Selective Self-tests/Logging not supported
As I understand it, the Offline_Uncorrectable attribute with value of 6 looks very bad since it seems to me that some data was lost 6 times. Am I missing something?
Also, the raw value of the Hardware_ECC_Recovered attribute looks huge, especially when compared to the Total_LBAs_Read attribute.
Should I worry about my SSD drive? Should I replace it?
For what it's worth, this is a TLC drive. I know they are less durable than the MLC (or the even better SLC) drives, but I did not expect data loss after only a few months.
Partial answer:
You have 6 Offline_uncorrectable (which should occur during offline-tests) and 6 Reallocated_Event_Count, but zero Reallocated_Sector_Ct. I read this as "tried to reallocate a block during an offline test six times, but failed each time".
The error message in the kernel log also says "auto reallocate failed", though this is an online event, not an offline one.
So:
1) Use smartctl to look at the result of the offline-tests, in particular at what time they were run, and if errors occured.
2) Use smartctl to start another offline test (overnight etc.). Look at the results.
3) You mentioned "some new read failures" in the comment. What are the block numbers for them? Is it always the same block?
If it's a single block that for some reason can't be auto-reallocated, I'd consider marking that block as bad in the file system (most file systems have that feature). That's usually not done today because all harddrives auto-reallocate, both mechanical and SSD, but the feature exists for exactly that purpose.
Though I'd be more worried that the auto-reallocate failed; that may mean that the (invisible) block that contains the reallocation-meta-information is bad.
I'm not sure what to do about that; if the vendor of the SDD has a special tool for download where you can adjust the low-level stuff (some vendors do), this may help.