Ubuntu Incorrectly Believes 100% of "/" Used, All Old Kernels & Large FilesRemoved, Won't Complete Upgrade Or Boot

Well there's your problem:

Filesystem       Size  Used Avail Use% Mounted on
/dev/nvme0n1p5    92G   87G     0 100% /

Your / location is full. The 5G of storage that appears unused is reserved by the system for logging and other core functions, meaning that you cannot use it without changing the reserve space for the filesystem ... which I strongly discourage when talking about /.

You will need to find out where all your storage space is being used and move/delete files accordingly. The typical place to start looking is /var/log. In here you can (probably) move/delete some of the archived .gz files. That could free up a good bit of space.

Alternatively, you can use du to identify where you are using the most storage:

sudo du / -h --max-depth=1

This will give you something that looks like:

8.0K    /media
84K     /root
4.0K    /mnt
1.7M    /run
16K     /lost+found
300K    /home
201M    /boot
17G     /var
8.1M    /etc
16K     /dev
22M     /opt
4.0K    /cdrom
0       /sys
4.0K    /srv
76K     /tmp
2.2G    /snap
0       /proc
65G     /data
3.5G    /usr
123G    /

From here, you can edit the / part of the du command to identify specific directories that may be using far more storage than you would like.


I solved the problem, but I did not answer the question, and created a new mystery.

  1. How I solved the problem: From the rescue shell of my ubuntu default kernel 5.11~~ I ran:

     # du / -h --max-depth=1
    

    I noticed that my /media folder weighed in at 36GB, which should be impossible as I had no external devices (usb) connected. First, I thought they were old mounts somehow locked in place and I tried umount /media/USB, but was told that they were not mounted.. since these are external devices and my data is safe, I went ahead and removed them.

     # rm -r /media/usb
    

    Now machine is happy. I was able to boot and complete my update/upgrades.

  2. But my original question remains. Even with the "ghost" mounted usb present, I still deleted over 2 gigs of data, while ubuntu only wanted 529 mb, what I deleted should have been more than enough to complete the update and boot my system? Why wasn't Ubuntu recognizing that I was deleting items?

    I used lsof | grep DEL and it returned no value... so I still don't have an explanation for why the space I deleted wasn't recognized.

  3. And the new mystery that I seem to have created is "Why were 'ghost' mounts of my USB's eating 36GB of space under "/"? I really would like to know this. Could my use of TimeShift somehow created the "ghost" mounts?

In any case @matigo's answer was partially useful, although it doesn't answer the question.