ZFS datasets wont mount after upgrade to ubuntu 21.10

So i recently upgraded from Ubuntu 21.04 to 21.10 and now my ZFS datasets wont mount, even manually using the forced option.

I have tried exporting the pool and re-importing the pool, but even trying it with the Forced options continue to fail.

root@daviserver:~# zfs mount -af
cannot mount 'ssdpool/sftpsync': Insufficient privileges
cannot mount 'ssdpool/sftpsync': Insufficient privileges
cannot mount 'ssdpool/docker': mount failed

syslog shows the following, but im not sure what it means.

Oct 17 13:19:21 daviserver kernel: [62900.992374] VFS: "mand" mount option not supported
Oct 17 13:19:21 daviserver kernel: [64689.402851] VFS: "mand" mount option not supported
Oct 17 13:19:21 daviserver kernel: [64689.402930] VFS: "mand" mount option not supported

I am currently scrubbing the pool to attempt to fix any errors.

root@daviserver:~# zpool status
  pool: ssdpool
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: scrub in progress since Sun Oct 17 13:25:30 2021
    683G scanned at 898M/s, 70.0G issued at 92.0M/s, 683G total
    0B repaired, 10.26% done, 01:53:36 to go
config:

    NAME                                                 STATE     READ WRITE CKSUM
    ssdpool                                              ONLINE       0     0     0
      mirror-0                                           ONLINE       0     0     0
        ata-Samsung_SSD_860_EVO_M.2_1TB_S415NB0KB06535B  ONLINE       0     0     0
        ata-Samsung_SSD_860_EVO_M.2_1TB_S415NB0KB07318V  ONLINE       0     0     0

errors: No known data errors

Solution 1:

I encountered this issue today with the same dmesg saying

[   21.385582] VFS: "mand" mount option not supported
[   21.411597] VFS: "mand" mount option not supported
[   21.478512] audit: type=1400 audit(1636396227.595:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libreoffice-senddoc" pid=3184 comm="apparmor_parser"
[   21.478525] audit: type=1400 audit(1636396227.595:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libreoffice-xpdfimport" pid=3186 comm="apparmor_parser"
[   21.478683] audit: type=1400 audit(1636396227.595:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=3174 comm="apparmor_parser"
[   21.479124] audit: type=1400 audit(1636396227.595:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=3175 comm="apparmor_parser"
[   21.479128] audit: type=1400 audit(1636396227.595:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=3175 comm="apparmor_parser"
[   21.479449] audit: type=1400 audit(1636396227.595:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=3178 comm="apparmor_parser"
[   21.479453] audit: type=1400 audit(1636396227.595:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=3178 comm="apparmor_parser"
[   21.479456] audit: type=1400 audit(1636396227.595:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=3178 comm="apparmor_parser"
[   21.480818] audit: type=1400 audit(1636396227.599:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libreoffice-oosplash" pid=3183 comm="apparmor_parser"
[   21.481815] audit: type=1400 audit(1636396227.599:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="virt-aa-helper" pid=3187 comm="apparmor_parser"
[   21.744450] VFS: "mand" mount option not supported
[   21.747138] VFS: "mand" mount option not supported
[   21.747190] VFS: "mand" mount option not supported
[   21.747345] VFS: "mand" mount option not supported
[   21.747378] VFS: "mand" mount option not supported

on my SMB server and I remembered that there was a recommended zfs option for ZFS SMB setup named nbmand for non-blocking mandatory locks. And all my zpool that won't mount has this option turned on.

sudo zfs get nbmand,mounted | grep -A1 ' on '

The mand mount option was removed by Aug 2021 and landed in 21.10's kernel, it should be ignored by mount command but ZFS's mount was not.

A sudo zfs inherit nbmand $POOL && sudo zfs mount -a brings back all zpools and all data.