Unable to upgrade zfsutils-linux on 20.04
So. I'm stuck. This is recent installation of Ubuntu 20.04. All disks are setup as ZFS filesystems.
⚡ root@docker /var zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
bpool 1,88G 506M 1,38G - - 0% 26% 1.00x ONLINE -
data 10,9T 1,76G 10,9T - - 0% 0% 1.00x ONLINE -
rpool 920G 7,61G 912G - - 0% 0% 1.00x ONLINE -
bpool is my boot partion, rpool my root. It looks like this:
⚡ root@docker /var zfs list
NAME USED AVAIL REFER MOUNTPOINT
bpool 505M 1,26G 96K /boot
bpool/BOOT 503M 1,26G 96K none
bpool/BOOT/ubuntu_u1x577 503M 1,26G 269M /boot
data 1,76G 10,5T 96K /data
data/home 110M 10,5T 110M /home
data/var 1,65G 10,5T 1,58G /var
data/var/lib 77,8M 10,5T 96K /var/lib
data/var/lib/docker 77,7M 10,5T 772K /var/lib/docker
data/var/lib/docker/3507eb58f488f64c7b9fc52fec17371e733dccf691d88a55ce23e4a732694f1d 84K 10,5T 76,5M legacy
data/var/lib/docker/66f1d3eb20e22d69da12e47e6e363c2c4ccfe71d9c9d58b46c4a27a0a143b354 5,02M 10,5T 11,1M legacy
data/var/lib/docker/7fac28892c840428c90e8a0eff4289b3791f816ea8eef6a0606d141fb2f83ada 65,5M 10,5T 76,5M legacy
data/var/lib/docker/cd2984a7ec9b4a4d4138d92c189d5c4c13fe4ace09aa88f3cd7603fd65005042 6,31M 10,5T 6,30M legacy
data/volumes 308K 10,5T 308K /data/volumes
rpool 7,61G 884G 96K /
rpool/ROOT 7,48G 884G 96K none
rpool/ROOT/ubuntu_u1x577 7,48G 884G 3,31G /
I've snipped off the rest, because I don't think it's important. If I run apt install I get the following:
⚡ root@docker /var apt install
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
3 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
INFO Requesting to save current system state
Successfully saved as "autozsys_apdldl"
Setting up zfsutils-linux (0.8.3-1ubuntu12) ...
zfs-import-scan.service is a disabled or a static unit, not starting it.
zfs-import-scan.service is a disabled or a static unit, not starting it.
Job for zfs-mount.service failed because a fatal signal was delivered to the control process.
See "systemctl status zfs-mount.service" and "journalctl -xe" for details.
invoke-rc.d: initscript zfs-mount, action "start" failed.
● zfs-mount.service - Mount ZFS filesystems
Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
Active: failed (Result: signal) since Fri 2020-04-24 23:51:39 CEST; 9ms ago
Docs: man:zfs(8)
Process: 71542 ExecStart=/sbin/zfs mount -a (code=killed, signal=SEGV)
Main PID: 71542 (code=killed, signal=SEGV)
Apr 24 23:51:38 docker systemd[1]: Starting Mount ZFS filesystems...
Apr 24 23:51:38 docker zfs[71542]: cannot mount '/var': directory is not empty
Apr 24 23:51:38 docker zfs[71542]: cannot mount '/home': mount failed
Apr 24 23:51:39 docker systemd[1]: zfs-mount.service: Main process exited, code=killed, status=11/SEGV
Apr 24 23:51:39 docker systemd[1]: zfs-mount.service: Failed with result 'signal'.
Apr 24 23:51:39 docker systemd[1]: Failed to start Mount ZFS filesystems.
dpkg: error processing package zfsutils-linux (--configure):
installed zfsutils-linux package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of zfs-initramfs:
zfs-initramfs depends on zfsutils-linux (>= 0.8.3-1ubuntu12); however:
Package zfsutils-linux is not configured yet.
dpkg: error processing package zfs-initramfs (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of zfs-zed:
zfs-zed depends on zfsutils-linux (>= 0.8.3-1ubuntu12); however:
Package zfsutils-linux is not configured yet.
dpkg: error processing package zfs-zed (--configure):
dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
No apport report written because the error message indicates its a followup error from a previous failure.
Errors were encountered while processing:
zfsutils-linux
zfs-initramfs
zfs-zed
INFO Updating GRUB menu
E: Sub-process /usr/bin/dpkg returned an error code (1)
So, it looks like zfsutils-linux is trying to remount /var and /home, but it can't because the directory isn't empty (because I'm assuming it's already mounted?).
Does anybody have any ideas where I can troubleshoot?
I figured out what is going on.
I have /var and /home as part of the data ZFS pool. Apparently at some point when it wasn't mounted, something wrote in /var and /home which means the data ZFS pool can't mount because as the message says cannot mount '/var': directory is not empty.
This would have been a little clearer if they had included the pool name in the error message.
If anybody else gets into this situation:
- You need to stop all the services on the box that could be writing to the directories affected.
- Then you need to move the directories out of the way.
mv /var /var.old - Now force zfs to mount the directories:
zfs mount -a - For each directory affected, copy the old files over the new
rsync -avz --delete /var.old/* /var/ - Cross your fingers (press your thumbs, or whatever culturally appropriate gesture fits)
- Reboot
I had the same problem (in my case, a few files existed in /home which prevented /home/username from a clean mount), but solved it differently and with less hassle:
-
Boot into recovery mode from GRUB and select Drop to root shell prompt
-
unmount all ZFS datasets:
zfs unmount -a -
verify that all pools have been unmounted
-
manually mount the problematic datasets:
zfs mount pool/home -
check for files that shouldn't be there and move those out of the way:
mv /home/username /home/username.old -
check whether mounting all ZFS datasets still produces errors
zfs mount -a -
if so, repeat, otherwise
reboot -
find out what to do with the files in
/home/username.old