Disk drive serial number reversed in /dev/disk/by-id/scsi-*

I'm using Ubuntu 20.04 with USB external drives and experiencing a rather odd bug. I've got a bunch of USB enclosures for hard disk drives here (or rather sold as external drives):

• Samsung Portable SSD T5 (mSATA to USB with UASP internally)

  S/N reported to kernel (as per dmesg & lsusb -v): 12345688ACF4

  (This is a bogus disk serial over USB, different from label and SMART data.)

• Samsung Portable SSD T7 (NVMe to USB with UASP internally)

  (Reports correct disk serial over USB; identical to SMART data and disk label.)

• StarTech USB 3.1 gen2 enclosure for SATA hard disk drive, with UASP

  (Reports correct disk serial over USB; identical to SMART data and disk label.)

• Ewent EW7017 USB to SATA dongle with UASP

  (Reports bogus disk serial 000000004BA8 over USB, it's always the same.)

• WD Elements USB hard disk drive without UASP

  (Reports bogus disk serial over USB.)

All these are single-disk to USB enclosures basically.

Their serial number reported over USB to the kernel is sometimes wrong, but that seems a hardware issue with them. It's not something we can solve and my question is about something else.

When adding a drive, a convenient link is created for them in /dev/disk/by-id/:

/dev/disk/by-id/
├── dm-name-dm_crypt-0 -> ../../dm-0
[...]
├── scsi-SSamsung_Portable_SSD_T5_4FCA88654321 -> ../../sdb
├── scsi-SSamsung_Portable_SSD_T5_4FCA88654321-part1 -> ../../sdb1
[...]

This is really cool, as it allows me to be sure I'm working on the correct drive rather than a dangerous assumption sdb would be the correct drive.

As you can see the string scsi-SSamsung_Portable_SSD_T5_4FCA88654321 contains two errors, though:

1. The manufacturer name is prepended with an S.
2. The serial number is reversed; 12345688ACF4 became 4FCA88654321.

The same happens with the T7 drive.

The StarTech enclosure does not have the serial reversion:

├── scsi-STOSHIBA_MQ01ABB200_1234ABCD -> ../../sdb

Same for the Ewent one:

├── scsi-SWDC_WD20_NPVX-00EA4T0_000000004BA8 -> ../../sdb

The Western Digital non-UASP drive does not suffer from any of the two problems:

└── usb-WD_Elements_1048_1234ABCD-0:0 -> ../../sdb

Also sometimes it's an i prepended instead of an S, depending on the manufacturer.

Why is this? What piece of software is responsible for these errors?

I am also using a more professional hot plug storage enclosure and this one works fine with those drives. So, it seems purely over USB or UASP that this goes bonkers.

Even weirder; when using the same hardware on Debian Linux 10/Buster, it works fine (ata instead of scsi by the way! and yes UASP is enabled scsi host4: uas in dmesg):

├── ata-TOSHIBA_MQ01ABB200_1234ABCD -> ../../sdb

So, is this an Ubuntu bug after all?

Solution 1:

Partial answer:

There are quite a few different ways to get the drive serial number. (Read the ATA, ATAPI and USB storage specs for details).

If you want to find out which particular way is used for the symbolic links, you need to dive into udev, find out which command is used to create those links (very likely blkid), find your version of blkid, and read the source code for it.

That said, the likely culprit for the odd behavior are the different USB-to-SATA bridge chips in your enclosures.

To verify that this chip is the culprit, take the real hard drive out of the closure, connect it via SATA to your mainboard, and see what serial you get.

Errors like duplicating the first letter (SSamsung) or reversing the serial are typical bugs in the firmware for those chips that no one bothered to correct (or maybe no one noticed).

So I had a quick look what it does on my system (old Devuan, which is Debian without systemd): The udev-rules are in

/lib/udev/rules.d/60-persistent-storage.rules

and they call scsi_id (and not blkid), which is in /lib/udev/scsi_id and not in one of the normal places for binaries.

I couldn't get it to produce output from the commandline, but the man page says

scsi_id queries a SCSI device via the SCSI INQUIRY vital product data (VPD) page 0x80 or 0x83 and uses the resulting data to generate a value that is unique across all SCSI devices that properly support page 0x80 or page 0x83.

and it goes on

Identifiers based on page 0x80 are prefixed by the character 'S', the SCSI vendor, the SCSI product (model) and then the the serial number returned by page 0x80.

So that explains your S, which is actually not a duplication of the first letter.

And as far as I know, the vital product page is not the way to get the manufacturer and serial information in most other parts of Linux, so I wouldn't be surprised if your serial reversal is an actual firmware bug either in the harddisk or the USB-SATA-bridge.

You can check this by installing sg3_utils and directly issuing the INQUIRY commands (with --page) to your device (try both in and out of the enclosure).

I don't have a variety of distributions installed, so I have no idea how udev differs on Ubuntu Focal or Debian with systemd.

Solution 2:

The reversed serial number comes from the SCSI-level SCSI-ATA Translation Layer of the USB device/enclosure:

# lsusb.py -i | grep 3-4
  3-4               174c:55aa 00 1IF  [USB 3.10,  5000 Mbps,   0mA] (ASMT ASM105x B087078B14DA00062434)
    3-4:1.0           (IF) 08:06:62 4EPs (Mass Storage) uas host7 (sdc)

# hdparm -I /dev/sdc | grep -i 'serial number'
        Serial Number:      B087078B14DA00062434

# sg_vpd -p 0x80 /dev/sdc
Unit serial number VPD page:
  Unit serial number: 43426000AD41B870780B

# sg_vpd -r -p 0x80 /dev/sdc | hexdump -C
00000000  00 80 00 14 34 33 34 32  36 30 30 30 41 44 34 31  |....43426000AD41|
00000010  42 38 37 30 37 38 30 42                           |B870780B|
00000018

# /usr/lib/udev/ata_id -x /dev/sdc | grep -i serial
ID_SERIAL=SPCC_Solid_State_Disk_B087078B14DA00062434
ID_SERIAL_SHORT=B087078B14DA00062434

# /usr/lib/udev/scsi_id -p 0x83 -g -x /dev/sdc | grep -i serial
ID_SERIAL=35000000000000001
ID_SERIAL_SHORT=5000000000000001
ID_SCSI_SERIAL=43426000AD41B870780B

# /usr/lib/udev/scsi_id -g -p 0x80 -x /dev/sdc | grep -i serial
ID_SERIAL=SSPCC_Solid_State_Disk_43426000AD41B870780B
ID_SERIAL_SHORT=43426000AD41B870780B

The reason that the enclosure vendors make the enclosure firmware perform such reversal is probably due to the fact that, in the earlier versions of the SCSI Primary Commands standard (e.g. SPC-2 revision 20), the PRODUCT SERIAL NUMBER field is require to hold the value such that The least significant ASCII character of the serial number shall appear as the last byte in the Data-In Buffer.

I'm not sure what's the correct interpretation / true intention of this statement, but at least the vendors probably interpreted it more or less like this:

1. B087078B14DA00062434 is considered as a hexadecimal number, i.e. 0x3433343236303030414434314238373037383042.
2. Since an ASCII character takes one-byte, the least significant ASCII character of the serial number would be 0x42 / 4.
3. Then they assume the statement implies that when value is read into Data-In Buffer for the command that requested the VPD page, it should be read from there all the way up til "the most significant ASCII character" of it (i.e. 0x34 / B).

However, as of the current-latest version / draft of SPC (e.g. SPC-6 revision 5), the requirement has changed. Now The PRODUCT SERIAL NUMBER field contains right-aligned ASCII data, and such field shall have any unused bytes at the start of the field (i.e., lowest offset) and the unused bytes shall be filled with ASCII space characters (i.e., code value 20h). It does not seem to have any direction (in the sense of instruction) for the "used bytes".

Therefore, it is hard, if possible at all, for anyone to say whether userland programs (e.g. sg_vpd in sg3_utils and scsi_id in systemd/udev) should "reverse it back" when they output the value from the VPD page.

The reason that you see the serial number in its correct order is the same as why you see that being so in corresponding outputs above: the source is not the SCSI Unit Serial Number VPD page. systemd/udev has other program/builtin to gather identification info from a block device:

• https://github.com/systemd/systemd-stable/tree/v249.2/src/udev/ata_id
• https://github.com/systemd/systemd-stable/blob/v249.2/src/udev/udev-builtin-usb_id.c

ata_id only works if the enclosure supports the ATA PASSTHROUGH SCSI command (and, of course, if the drive behind is an ATA drive). See the SAT standard for more information.

It is unknown to me that why your system(d) / udev would end up making use of scsi_id for the device. As per the 60-persistent-storage.rules in recent versions of systemd, I'm not see it being prioritized over ata_id or the usb_id builtin. Check the one you have in {{/usr,}/lib,/etc}/udev/rules.d/ to see if it differs. I'm not familiar with Ubuntu to tell why anyway.

Btw, the S is prepended by scsi_id when your enclosure does not support VPD page 0x83 (I'm not sure if it still makes / ever made any sense; the code dates back to when the program was introduced to udev, before that was even merged to into systemd):

• https://github.com/systemd/systemd-stable/blob/v249.2/src/udev/scsi_id/scsi_serial.c#L724

Not sure about i or 0.

Also see these for more details about the heuristics:

• https://github.com/systemd/systemd-stable/blob/v249.2/src/udev/scsi_id/scsi_serial.c#L853
• https://github.com/systemd/systemd-stable/blob/v249.2/src/udev/scsi_id/scsi_serial.c#L559