GRUB Bootloader with root LUKS encryption: Only grub shell

I would appreciate help and hints to solve a problem with the GRUB bootloader. I have a system with root LUKS encryption. Means that except the partition with the bootloader, the whole system is in an encrypted LUKS container. I am unsure where the possible problems are, so I tell you some more details.

After a system update, GRUB now hangs in the GRUB shell (not rescue mode).

I can however start manually with:

insmod cryptodisk
insmod luks
insmod lvm
insmod ext2
cryptomount -a
set root=(lvm/system-root)
linux (lvm/system-boot)/vmlinuz-[...]-generic root=/dev/system/root
initrd (lvm/system-boot)/initrd.img-[...]-generic

Running lsblk on my system looks like this:

nvme0n1           259:0    0 953,9G  0 disk
├─nvme0n1p1       259:1    0   450M  0 part
├─nvme0n1p2       259:2    0   100M  0 part  /boot/efi
├─nvme0n1p3       259:3    0    16M  0 part
├─nvme0n1p4       259:4    0  97,9G  0 part
└─nvme0n1p5       259:5    0 839,4G  0 part
  └─system        253:0    0 839,4G  0 crypt
    ├─system-boot 253:1    0   512M  0 lvm   /boot
    ├─system-swap 253:2    0  16,1G  0 lvm   [SWAP]
    └─system-root 253:3    0 822,8G  0 lvm   /

p1, p3, p4 belong to an installed windows 10. p2 is the fat32 EFI system partition p5 is the luks encrypted system with boot, swap and root partition.

Maybe Grub cannot load the grub.cfg?

I thought Grub does not find the config file in the p2 partition in /boot/efi/grub/grub.cfg.

But if this would be the case, the manual execution of the config file should succeed. So I executed (used auto-completion, so the path should be right) in the grub shell at startup:

configfile (hd0,gpt2)/grub/grub.cfg

But i just get a cleared grub shell (without any errors).

Why I thought grub does not find the grub.cfg is because of the content of the grub config file in the EFI folder (which gets loaded first as I understood corecctly): /boot/efi/EFI/ubuntu/grub.cfg:

search.fs_uuid db041d94-37fa-42a6-b2f3-87572a38f23c root lvmid/uWUnvj-b70J-Gfam-Tfr3-87I3-GQAP-bL2Lym/gEpImK-yuWK-W6Ip-v5KF-ZGKm-8GZS-ALhXxF 
set prefix=($root)'/grub'
configfile $prefix/grub.cfg

Here the UUID(list at the end) it looks for is the one of theencrypted boot partition. This seems weired: Shouldn't it be visible for grub till I entered the password?.

But as my command to load the config file manually did not work I guess this is not the issue for my problem.

Checking /etc/default/grub

So it could be my configuration file: I cannot judge how right this file is, so here is the summarized content of the file:

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_ENABLE_CRYPTODISK=y

GRUB_CMDLINE_LINUX="cryptdevice=UUID=8d82f654-43fd-40ba-a185-1ead48838f54:system root=/dev/system/system-root resume=/dev/system/system-swap ro"

GRUB_PRELOAD_MODULES="luks cryptodisk lvm ext2"

To me this does not look too bad. The UUID of the cryptdevice looks fine. I tried changed the "/dev/system/system-root" to "/dev/system/system" as this is the path while running the system, but without any effect. i also did run update-grup and grub-install.

In addition I installed grub-emu (emulator for the grub bootloader). This did show me at least the gurb menu. When selecting the entry to start ubuntu I get at least some errors:

error: no such cryptodisk found.
error: no such device: db041d94-37fa-42a6-b2f3-87572a38f23c.
error: can't find command `linux'.
error: can't find command `initrd'.

What would help me

Of course point me to the problem and telling me the fix. As I now have to deal with it, i am curious about the boot process with grub:
I do not understand the content of the /boot/efi/EFI/ubuntu/grub.cfg.
What is the content of ($root) here? And why is it looking for an encrypted partition?
Where is it determined what content is in this file?
Why does it not show me at least the grub menu even if there are errors that would prevent the system to boot?

Thank you very much, Ecco

Additional content

The content of /boot/efi/grub/grub.cfg can be found at https://pastebin.com/D1X7jPsS

UUIDs (i omitted the windows specific partitions)

/dev/nvme0n1p2: UUID="2885-45AC" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID="1043df12-d11d-4023-bf6a-6acc2919c06f"
/dev/nvme0n1p5: UUID="8d82f654-43fd-40ba-a185-1ead48838f54" TYPE="crypto_LUKS" PARTLABEL="system" PARTUUID="2457b24a-7695-4737-ae46-5687aca3ab01"

UUIDs of the LUKS container

/dev/mapper/system: UUID="X7XHJG-pQ21-edgj-d73H-kK8Z-DxM8-eXBCh6" TYPE="LVM2_member"
/dev/mapper/system-root: LABEL="root" UUID="07e1d1e4-cd00-42dd-96cf-4f99864d0b7f" TYPE="ext4"
/dev/mapper/system-swap: UUID="02664e3e-3ba7-4a03-8b03-eb0335ca40f9" TYPE="swap"
/dev/mapper/system-boot: LABEL="boot" UUID="db041d94-37fa-42a6-b2f3-87572a38f23c" TYPE="ext4"

Finally it works (again): The cmd refreshgrub did the trick This command changed the content of the file /boot/efi/EFI/ubuntu/grub.cfg to:

search.fs_uuid 2885-45AC root 
set prefix=($root)'/EFI/ubuntu/grub'
configfile $prefix/grub.cfg

This makes more sense than before: Searching for the unencrypted partition. But now the bigger grub.cfg file is stored in another folder, which is ok.

It was now showing the menu, but the boot failed, I had to change in the /etc/default/grub.cfg the following line:

GRUB_CMDLINE_LINUX="cryptdevice=UUID=8d82f654-43fd-40ba-a185-1ead48838f54:system root=/dev/system/root resume=/dev/system/swap ro"