How to preserve drive device paths for AWS EC2 instance between reboots

I rebooted a m5d.xlarge EC2 instance running Ubuntu 16.04.5 LTS (xenial) using shutdown -r.

Upon reboot, various drive mounts defined in /etc/fstab did not line up with the device assignments given to them initially.

This caused various services to fail, which rely upon the drive mount with the assigned name containing specific data associated with that name.

In this case, /data needed to contain what is normally expected to be in /data, and not some other drive mount, like /foo or /bar or /whatever-else.

Running lsblk helped with manually redefining the /etc/fstab file to reflect the new device assignments and bringing services back online.

Problem: My worry is that rebooting will randomly reassign device labels to drive mounts, and this problem could or will occur again when the server requires rebooting.

Question: How do I ensure that drive mounts and their respective device paths are preserved between reboots, without needing manual (re)intervention?

Edit: The c5_m5_checks_script.sh script referenced in the answer provided below returns the following information:

# ./c5_m5_checks_script.sh 
------------------------------------------------

OK     NVMe Module is installed and available on your instance


ERROR  NVMe Module is not loaded in the initramfs image.
        - Please run the following command on your instance to recreate initramfs:
        # sudo update-initramfs -c -k all


OK     ENA Module with version 2.0.3K is installed and available on your instance


OK     fstab file looks fine and does not contain any device names.

------------------------------------------------

I don't understand the last OK statement, as the fstab file contains device names. Or Amazon's use of the term "device names" means something different from the device names in the fstab file, perhaps?

To give a specific and concrete example of the nature of the problem, I have an /etc/fstab that looks like this:

...
/dev/nvme2n1    /staging        ext4    defaults,nofail 0       0
/dev/nvme3n1    /data   ext4    defaults,nofail 0       0
...

However, lsblk shows that these two volumes are mounted to each other's device ID:

# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
...
nvme2n1     259:0    0     1T  0 disk /staging
nvme3n1     259:3    0   512G  0 disk /data

In reality, nvme2n1 is the 1T volume /data, and nvme3n1 is the 512G volume /staging.

Something unknown happens between reboots which reassigns the underlying drives to a permutation of the available device IDs.

The only way I know to fix this is to manually umount these two volumes, edit /etc/fstab, and mount them again.

Bounty question: Is there a permanent fix that would let me use a different and permanent identifier in /etc/fstab, which persists consistently between reboots?

Solution 1:

AWS has a tool to convert EBS volume device names in /etc/fstab to their UUID values: c5_m5_checks_script.sh

The script also checks that the NVMe module is loaded in your OS image, but your newer instance type would not have launched if that was not present.

Solution 2:

Why not using LABEL= or UUID= in /etc/fstab?

You can use blkid to find out corresponding labels and UUIDs. Even more info in man fstab of course:

It's also possible to use PARTUUID= and PARTLABEL=. These partitions identifiers are supported for example for GUID Partition Table (GPT).

See mount(8), blkid(8) or lsblk(8) for more details about device identifiers.

Note that mount(8) uses UUIDs as strings. The string representation of the UUID should be based on lower case characters.