Exit read-only file system after system crash and hard reset (20.04.2 Server, VM)
Yesterday my system crashed forcing me to do a hard reset on my host machine whilst running 2 virtual machines. The first machine survived and the second has been put into a read only mode. After logging in to the VM I am presented with the following messages.
Unable to setup logging. [Errno 30] Read-only file system: '/var/log/landscape/sysinfo.log' run-parts: /etc/update-mot.d/50-landscape-sysinfo exited with return code 1 /usr/lib/ubuntu-release-upgrader/release-upgrade-motd: 31: cannot create /var/lib/ubuntu-release-upgrader/release-upgrade-available: Read-only file system mktemp: failedd to create file via template '/var/lib/update-notifier/tmp.XXXXXXXXXX': Read-only file system run-parts: /etc/update-motd.d/95-hwe-eol exited with return code 1 /usr/lib/update-notifier/update-motd-fsck-at-reboot: 38: cannot create /var/lib/update-notifier/fsck-at-reboot: Read-only file system
I have checked the disk image file from the host, which in this case is a qcow2 file which I use in tandem with KVM.
sudo qemu-img check ubuntu204.qcow2
Result of which is:
No errors were found on the image. 245760/245760 = 100.00% allocated, 0.00% fragmented, 0.00% compressed clusters Image end offset: 16108814336
I have read that this is a forced state to protect the file system and that if I scan the disk and everything is OK I can just exit read-only mode, How would I do that with the following partition table on tty in read only mode?
loop0 0
loop1 0
loop2 0
loop3 0
loop4 0
loop5 0
|-sda 0 15G 0 disk
|-sda1 0 1M 0 part
|-sda2 0 1G 0 part /boot
|-sda3 0 14G 0 part
|-ubuntu--vg-ubuntu--lv 0 14G 0 lvm /
|-sr0 1 1024M 0 rom
From the warnings I suspect there may be a problem with unattended-upgrades I would also be interested in any further input regarding further checks you would recommend to ensure the integrity of both machines.
Solution 1:
How to recover a qcow2 disk image file using fsck from the host
Load network block device module:
sudo modprobe nbd max_part=8
Poweroff machine:
sudo virsh destroy virtual-machine
Connect disk image:
sudo qemu-nbd --connect=/dev/nbd0 /var/lib/libvirt/images/virtual-machine.qcow2
Check disk: (Disk is now mounted on host you can see it in nautilus/explorer)
sudo fsck /dev/nbd0p2
For me i had two extra drives mounted which I just scanned both nbd-dm3 and nbd0p2, here is an example of expected output which I just answered yes to all.
fsck from util-linux 2.25.2
e2fsck 1.42.12 (29-Aug-2014)
/dev/nbd0p1: recovering journal
/dev/nbd0p1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inodes that were part of a corrupted orphan linked list found. Fix<y>? yes
Inode 274 was part of the orphaned inode list. FIXED.
Inode 132276 was part of the orphaned inode list. FIXED.
Deleted inode 142248 has zero dtime. Fix<y>? yes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -603674 -623174 +(689342--689343)
Fix<y>? yes
Free blocks count wrong for group #18 (15076, counted=15077).
Fix<y>? yes
Free blocks count wrong for group #19 (11674, counted=11675).
Fix<y>? yes
Free blocks count wrong (632938, counted=670871).
Fix<y>? yes
Inode bitmap differences: -274 -132276 -142248
Fix<y>? yes
Free inodes count wrong for group #0 (52, counted=53).
Fix<y>? yes
Free inodes count wrong for group #16 (99, counted=100).
Fix<y>? yes
Free inodes count wrong for group #17 (519, counted=520).
Fix<y>? yes
Free inodes count wrong (204392, counted=204599).
Fix<y>? yes
/dev/nbd0p1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/nbd0p1: 101833/306432 files (0.2% non-contiguous), 553321/1224192 blocks
Disconnect device:
sudo qemu-nbd --disconnect /dev/nbd0
/dev/nbd0 disconnected
Start your machine and check if it worked:
virsh start virtual-machine
Source: https://gist.github.com/islander/8517685e3a9f7e0e1d458033710f0893