Landing in systemd emergency prompt after lengthy boot repair process. Problem: Missing FS in fstab flile

Solution 1:

So I actually figured out the issue while I was writing the question. As you can see from what I wrote in the beginning, it was a very long process (I had been working on it for about 2 days before I got to the point of wanting to ask for help).

If you look at the very end of the Q, I had received this message from dmesg during the boot process:

[FAILED] Failed to mount /boot
See 'systemctl status boot.mount' for details.

So, of course I tried systemctl status boot.mount to see what it said, but it said boot.mount is active (green), it's loaded and functioning properly, even though /boot was empty unless I manually mounted /dev/sda2 (which is exactly the opposite of what I would expect).

So I started thinking something might be wrong with the service. I disabled boot.mount even though it said it was working properly:

systemctl disable --now boot.mount

I tried to re-enable it, but got an error:

systemctl enable --now boot.mount
Failed to enable unit: Unit /run/systemd/generator/boot.mount is transient or generated

OK, that makes sense, it's triggered through the boot process and cannot be invoked through a user command. So I tried to re-mount all devices with:

mount -a

And saw that there was an error in the /etc/fstab file:

error: rw,relatime is not a valid file system

(or something to that effect).

The key here is, if I hadn't tried mounting the filesystem manually, I would have never received that feedback. The error message from mount -a one gets when fstab contains improper syntax is incredibly helpful. A lot more helpful than:

[FAILED] Failed to mount /boot
See 'systemctl status boot.mount' for details.

... and then seeing a "working" systemd unit for boot.mount when /boot is not mounting (even though it did get me to the right place eventually).

So I edited the fstab and entered the filesystem info for the /boot partition that failed to mount, then I re-ran mount -a (which essentially does the same thing as boot.mount) and got a positive response.

Now the two partitions are mounting properly after a reboot, and all is good in the land of horseradish and marmalade.

If this does not address any of your issues, here are some additional notes of the process I went through before getting to the point above where I was looking for help (feel free to stop reading after you get to your problem):

The original issue I was having two days ago was the system trying to boot from kernels no longer on the system. So, after booting with the live CD, I deleted the /boot folder's contents (where all the initrd files are located).

I figured I would just re-create the initramfs using update-initramfs -c -k all from the current kernels I had installed, but then I learned that I could not re-create the config or System.map files with depmod alone. This turned out to be a little more troublesome than I had bargained for.

I found the easiest way to re-generate or acquire all these files is to:

1. delete all contents of /boot,
2. uninstall any linux-image, linux-header and linux-modules files I had no intention of using,
3. delete all residual directories in /usr/lib/modules, and then
4. re-install linux-image, linux-modules and linux-headers files I intended on using (the most current generic two versions)

Note: Re-installing these 3 types of files all at the same time was how I managed to get the /boot/System.map and /boot/config files back - before only re-installing the linux-image files did not do it. It's possible they're included with modules (modules would make sense), or headers packages, but this is what worked for me.

5. Then I ran update-grub after re-installing those files and confirming /boot was populated correctly.
6. I also ran bootctl install and /etc/kernel/postinst.d/zz-udpate-systemd-boot, so I would have systemd-boot installed as a fallback.

At one point after a reboot, I had to re-configure system.target to multi-user.target instead of graphical.target, probably due to having chrooted with all those mounts in a graphical live CD to run the boot-repair program a couple days ago, which requires graphics (and I believe /dev/pts /tmp and /run were required to get display :0.0 to work):

systemctl set-default multi-user.target

Ok that's about it. Hope this helps someone.