Kernel Modules Created by DKMS Show "Exec format error" (after update to Ubuntu 21.04)
Hi I'm in a weird situation. The update to Hirsute seemed just fine until I plugged my Dockingstation in the Laptop and noticed that I wouldn't be able to use my external monitors. Narrowing down the cause led me to DKMS failing to produce modules for my kernel, as also the NVidia drivers and VirtualBox were not usable anymore.
What I found out is that GCC and BinUtils on the system should match the versions used to create the kernel.
So dmesg output contains:
sudo dmesg
Linux version 5.11.0-18-generic (buildd@lgw01-amd64-049) (gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0, GNU ld (GNU Binutils for Ubuntu) 2.36.1) #19-Ubuntu SMP Fri May 7 14:22:03 UTC 2021 (Ubuntu 5.11.0-18.19-generic 5.11.17)
GCC version is:
gcc --version
gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
And binutils is:
sudo apt show binutils
Version: 2.36.1-6ubuntu1
So everything seems just fine, even when reinstalling for example virtualbox-dkms:
sudo apt install virtualbox-dkms
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
libcg libcggl libnvidia-decode-460:i386 libnvidia-encode-460:i386 libnvidia-fbc1-460:i386 libnvidia-gl-460:i386 libnvidia-ifr1-460:i386 libpython2-stdlib libpython2.7-minimal libpython2.7-stdlib libxnvctrl0
nvidia-prime nvidia-settings nvidia-utils-460 python-is-python2 python2 python2-minimal python2.7 python2.7-minimal screen-resolution-extra xserver-xorg-video-nvidia-460
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
dkms
Suggested packages:
menu
The following NEW packages will be installed:
dkms virtualbox-dkms
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 761 kB of archives.
After this operation, 5.983 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://de.archive.ubuntu.com/ubuntu hirsute/main amd64 dkms all 2.8.4-4 [68,3 kB]
Get:2 http://de.archive.ubuntu.com/ubuntu hirsute/multiverse amd64 virtualbox-dkms amd64 6.1.18-dfsg-5 [692 kB]
Fetched 761 kB in 0s (2.382 kB/s)
Selecting previously unselected package dkms.
(Reading database ... 237715 files and directories currently installed.)
Preparing to unpack .../archives/dkms_2.8.4-4_all.deb ...
Unpacking dkms (2.8.4-4) ...
Selecting previously unselected package virtualbox-dkms.
Preparing to unpack .../virtualbox-dkms_6.1.18-dfsg-5_amd64.deb ...
Unpacking virtualbox-dkms (6.1.18-dfsg-5) ...
Setting up dkms (2.8.4-4) ...
Setting up virtualbox-dkms (6.1.18-dfsg-5) ...
Loading new virtualbox-6.1.18 DKMS files...
Building for 5.11.0-18-generic
Building initial module for 5.11.0-18-generic
Secure Boot not enabled on this system.
Done.
vboxdrv.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/5.11.0-18-generic/updates/dkms/
vboxnetadp.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/5.11.0-18-generic/updates/dkms/
vboxnetflt.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/5.11.0-18-generic/updates/dkms/
depmod....
DKMS: install completed.
But now I see that the virtualbox.service is not running:
systemctl status virtualbox.service
● virtualbox.service - LSB: VirtualBox Linux kernel module
Loaded: loaded (/etc/init.d/virtualbox; generated)
Active: failed (Result: exit-code) since Sat 2021-06-12 20:05:44 CEST; 5min ago
Docs: man:systemd-sysv-generator(8)
Process: 2331 ExecStart=/etc/init.d/virtualbox start (code=exited, status=1/FAILURE)
Jun 12 20:05:43 icy systemd[1]: Starting LSB: VirtualBox Linux kernel module...
Jun 12 20:05:43 icy virtualbox[2331]: * Loading VirtualBox kernel modules...
Jun 12 20:05:44 icy virtualbox[2331]: * No suitable module for running kernel found
Jun 12 20:05:44 icy virtualbox[2331]: ...fail!
Jun 12 20:05:44 icy systemd[1]: virtualbox.service: Control process exited, code=exited, status=1/FAILURE
Jun 12 20:05:44 icy systemd[1]: virtualbox.service: Failed with result 'exit-code'.
Jun 12 20:05:44 icy systemd[1]: Failed to start LSB: VirtualBox Linux kernel module.
And the reason seems to be the module itself:
sudo modprobe vboxdrv
modprobe: ERROR: could not insert 'vboxdrv': Exec format error
So finally the question is what can I do to make DKMS modules run again?
Update 1
As requested here is also the output of modinfo:
modinfo vboxdrv
filename: /lib/modules/5.11.0-18-generic/updates/dkms/vboxdrv.ko
version: 6.1.18_Ubuntu r142142 (0x00300000)
license: GPL
description: Oracle VM VirtualBox Support Driver
author: Oracle Corporation
srcversion: 5EB879B583F3535AB3CD593
depends:
retpoline: Y
name: vboxdrv
vermagic: 5.11.0-18-generic SMP mod_unload modversions
sig_id: PKCS#7
signer: icy Secure Boot Module Signature key
sig_key: 23:63:30:57:34:DB:4B:33:E5:70:6F:95:8E:AD:BC:C3:58:DA:A9:35
sig_hashalgo: sha512
signature: 52:41:92:C6:00:84:D2:AC:7C:CB:B2:18:72:35:91:2D:49:AE:5C:CB:
B4:E6:48:00:63:B2:E9:30:3A:68:D8:EC:CA:58:B4:14:0F:4D:4A:C2:
83:BA:70:3A:6E:B8:F2:07:7D:8A:5B:F0:03:C8:83:61:FF:F9:1B:0D:
ED:53:85:FF:6E:4F:C2:8A:1B:B5:4C:DC:79:13:00:D9:F8:87:58:19:
3E:F6:C4:BC:93:5D:9F:6C:60:42:B0:55:8E:A1:B5:9E:50:EB:0C:BB:
92:42:82:BE:F1:24:89:C3:5C:CB:3E:2B:51:66:57:00:62:55:63:2E:
D6:4A:10:47:66:4D:33:F6:0F:DF:02:A3:17:1E:34:7D:BF:64:99:88:
E3:D2:18:DA:B9:39:13:BC:19:28:E3:6A:F4:51:94:41:35:E5:F0:C5:
5E:61:A2:11:C3:1C:D4:4B:7C:09:64:41:C9:A1:D6:73:A2:5C:E9:FF:
21:E4:A3:B7:85:5C:17:AE:75:1A:06:41:3F:63:3A:69:36:2F:3C:C9:
4B:8B:65:FF:92:1B:00:67:43:37:32:91:9A:9E:A6:C9:42:EB:1A:18:
A4:47:89:B2:E0:02:B8:C6:6F:DE:B4:D8:40:34:30:70:84:85:4F:C1:
1D:B2:BA:BE:4A:15:73:6C:C0:99:DE:06:8C:E6:89:74
parm: force_async_tsc:force the asynchronous TSC mode (int)
Update 2
dpkg:
dpkg -l *virtualbox*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============================-=========================-============-============================================================
ii virtualbox 6.1.18-dfsg-5 amd64 x86 virtualization solution - base binaries
un virtualbox-2.0 <none> <none> (no description available)
un virtualbox-2.1 <none> <none> (no description available)
un virtualbox-2.2 <none> <none> (no description available)
un virtualbox-3.0 <none> <none> (no description available)
un virtualbox-3.1 <none> <none> (no description available)
un virtualbox-3.2 <none> <none> (no description available)
un virtualbox-4.0 <none> <none> (no description available)
un virtualbox-4.1 <none> <none> (no description available)
un virtualbox-4.2 <none> <none> (no description available)
un virtualbox-4.3 <none> <none> (no description available)
un virtualbox-5.0 <none> <none> (no description available)
un virtualbox-5.1 <none> <none> (no description available)
un virtualbox-5.2 <none> <none> (no description available)
un virtualbox-6.0 <none> <none> (no description available)
rc virtualbox-6.1 6.1.22-144080~Ubuntu~eoan amd64 Oracle VM VirtualBox
ii virtualbox-dkms 6.1.18-dfsg-5 amd64 x86 virtualization solution - kernel module sources for dkms
un virtualbox-guest-additions-iso <none> <none> (no description available)
un virtualbox-guest-dkms <none> <none> (no description available)
un virtualbox-guest-modules <none> <none> (no description available)
un virtualbox-modules <none> <none> (no description available)
un virtualbox-ose <none> <none> (no description available)
ii virtualbox-qt 6.1.18-dfsg-5 amd64 x86 virtualization solution - Qt based user interface
ii virtualbox-source 6.1.18-dfsg-5 amd64 x86 virtualization solution - kernel module source
usr/src:
ls -al /usr/src/
total 900
drwxr-xr-x 29 root root 4096 Jun 12 20:34 .
drwxr-xr-x 14 root root 4096 Apr 23 2020 ..
drwx------ 2 nm-openvpn pulse 4096 Jun 18 2019 evdi-5.2.14
drwxr-xr-x 3 root root 4096 Mai 14 2020 libdvd-pkg
drwxr-xr-x 24 root root 4096 Jun 5 13:55 linux-headers-5.11.0-18
drwxr-xr-x 7 root root 4096 Jun 5 14:13 linux-headers-5.11.0-18-generic
drwxr-xr-x 20 root root 4096 Jun 5 2020 linux-headers-5.4.0-29
drwxr-xr-x 4 root root 4096 Jun 5 2020 linux-headers-5.4.0-29-generic
drwxr-xr-x 3 root root 4096 Jun 16 2020 linux-headers-5.4.0-31
drwxr-xr-x 20 root root 4096 Jul 6 2020 linux-headers-5.4.0-33
drwxr-xr-x 4 root root 4096 Jul 6 2020 linux-headers-5.4.0-33-generic
drwxr-xr-x 20 root root 4096 Jul 6 2020 linux-headers-5.4.0-37
drwxr-xr-x 4 root root 4096 Jul 6 2020 linux-headers-5.4.0-37-generic
drwxr-xr-x 20 root root 4096 Aug 1 2020 linux-headers-5.4.0-39
drwxr-xr-x 4 root root 4096 Aug 1 2020 linux-headers-5.4.0-39-generic
drwxr-xr-x 20 root root 4096 Sep 11 2020 linux-headers-5.4.0-40
drwxr-xr-x 4 root root 4096 Sep 11 2020 linux-headers-5.4.0-40-generic
drwxr-xr-x 20 root root 4096 Sep 16 2020 linux-headers-5.4.0-42
drwxr-xr-x 4 root root 4096 Sep 16 2020 linux-headers-5.4.0-42-generic
drwxr-xr-x 20 root root 4096 Sep 27 2020 linux-headers-5.4.0-45
drwxr-xr-x 4 root root 4096 Sep 27 2020 linux-headers-5.4.0-45-generic
drwxr-xr-x 20 root root 4096 Okt 25 2020 linux-headers-5.4.0-48
drwxr-xr-x 4 root root 4096 Okt 25 2020 linux-headers-5.4.0-48-generic
drwxr-xr-x 20 root root 4096 Nov 3 2020 linux-headers-5.4.0-51
drwxr-xr-x 4 root root 4096 Nov 3 2020 linux-headers-5.4.0-51-generic
drwxr-xr-x 20 root root 4096 Nov 3 2020 linux-headers-5.4.0-52
drwxr-xr-x 4 root root 4096 Nov 3 2020 linux-headers-5.4.0-52-generic
drwxr-xr-x 7 root root 4096 Jun 12 19:57 nvidia-460.80
drwxr-xr-x 11 root root 4096 Jun 12 20:34 virtualbox-6.1.18
-rw-r--r-- 1 root root 802665 Apr 8 20:43 virtualbox.tar.bz2
modules:
ls -al /lib/modules/$(uname -r)/updates/dkms
total 680
drwxr-xr-x 2 root root 4096 Jun 12 20:34 .
drwxr-xr-x 3 root root 4096 Jun 12 20:34 ..
-rw-r--r-- 1 root root 620706 Jun 12 20:34 vboxdrv.ko
-rw-r--r-- 1 root root 17474 Jun 12 20:34 vboxnetadp.ko
-rw-r--r-- 1 root root 44378 Jun 12 20:34 vboxnetflt.ko
dkms status:
dkms status
virtualbox, 6.1.18, 5.11.0-18-generic, x86_64: installed
Update 3
The following command showed no results:
grep -i virtualbox /etc/apt/sources.list
In sources.list.d:
grep -i virtualbox /etc/apt/sources.list.d/*.list
/etc/apt/sources.list.d/virtualbox.list:deb [arch=amd64] https://download.virtualbox.org/virtualbox/debian hirsute contrib
Well I will delete this file since installing virtualbox-6.1 is not working anyways. But I think the root cause is still DKMS since I have the same problem with for example evdi.ko when installing from DisplayLink driver from synaptics.com.
Synaptics shows me virtualbox and virtualbox-6.1 only. After removing /etc/apt/sources.list.d/virtualbox.list only virtualbox is left.
Reinstalled dkms and virtualbox. Still the same problem.
Update 4
Turns out I cannot even build a kernel module without dkms, so something is really bad.
Tried this tutorial Writing a Simple Linux Kernel Module
Here is what I got:
user@icy:~/src/lkm-example$ make
make -C /lib/modules/5.11.0-18-generic/build M=/home/user/src/lkm-example modules
make[1]: Entering directory '/usr/src/linux-headers-5.11.0-18-generic'
CC [M] /home/user/src/lkm-example/lkm-example/lkm_example.o
MODPOST /home/user/src/lkm-example/lkm-example/Module.symvers
CC [M] /home/user/src/lkm-example/lkm-example/lkm_example.mod.o
LD [M] /home/user/src/lkm-example/lkm-example/lkm_example.ko
make[1]: Leaving directory '/usr/src/linux-headers-5.11.0-18-generic'
user@icy:~/src/lkm-example$ sudo insmod lkm_example.ko
[sudo] password for user:
insmod: ERROR: could not insert module lkm_example.ko: Invalid module format
user@icy:~/src/lkm-example$ modinfo lkm_example.ko
filename: /home/user/src/lkm-example/lkm_example.ko
version: 0.01
description: A simple example Linux module.
author: Robert W. Oliver II
license: GPL
srcversion: 0EC24E04A8469A9DB84342D
depends:
retpoline: Y
name: lkm_example
vermagic: 5.11.0-18-generic SMP mod_unload modversions
Update 5 - Current State
user@icy:~$ modinfo vboxdrv
filename: /lib/modules/5.11.0-18-generic/updates/dkms/vboxdrv.ko
version: 6.1.18_Ubuntu r142142 (0x00300000)
license: GPL
description: Oracle VM VirtualBox Support Driver
author: Oracle Corporation
srcversion: 5EB879B583F3535AB3CD593
depends:
retpoline: Y
name: vboxdrv
vermagic: 5.11.0-18-generic SMP mod_unload modversions
sig_id: PKCS#7
signer: icy Secure Boot Module Signature key
sig_key: 23:63:30:57:34:DB:4B:33:E5:70:6F:95:8E:AD:BC:C3:58:DA:A9:35
sig_hashalgo: sha512
signature: 52:41:92:C6:00:84:D2:AC:7C:CB:B2:18:72:35:91:2D:49:AE:5C:CB:
B4:E6:48:00:63:B2:E9:30:3A:68:D8:EC:CA:58:B4:14:0F:4D:4A:C2:
83:BA:70:3A:6E:B8:F2:07:7D:8A:5B:F0:03:C8:83:61:FF:F9:1B:0D:
ED:53:85:FF:6E:4F:C2:8A:1B:B5:4C:DC:79:13:00:D9:F8:87:58:19:
3E:F6:C4:BC:93:5D:9F:6C:60:42:B0:55:8E:A1:B5:9E:50:EB:0C:BB:
92:42:82:BE:F1:24:89:C3:5C:CB:3E:2B:51:66:57:00:62:55:63:2E:
D6:4A:10:47:66:4D:33:F6:0F:DF:02:A3:17:1E:34:7D:BF:64:99:88:
E3:D2:18:DA:B9:39:13:BC:19:28:E3:6A:F4:51:94:41:35:E5:F0:C5:
5E:61:A2:11:C3:1C:D4:4B:7C:09:64:41:C9:A1:D6:73:A2:5C:E9:FF:
21:E4:A3:B7:85:5C:17:AE:75:1A:06:41:3F:63:3A:69:36:2F:3C:C9:
4B:8B:65:FF:92:1B:00:67:43:37:32:91:9A:9E:A6:C9:42:EB:1A:18:
A4:47:89:B2:E0:02:B8:C6:6F:DE:B4:D8:40:34:30:70:84:85:4F:C1:
1D:B2:BA:BE:4A:15:73:6C:C0:99:DE:06:8C:E6:89:74
parm: force_async_tsc:force the asynchronous TSC mode (int)
user@icy:~$ dpkg -l *virtualbox*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============================-=============-============-==========================================>
ii virtualbox 6.1.18-dfsg-5 amd64 x86 virtualization solution - base binaries
un virtualbox-2.0 <none> <none> (no description available)
un virtualbox-2.1 <none> <none> (no description available)
un virtualbox-2.2 <none> <none> (no description available)
un virtualbox-3.0 <none> <none> (no description available)
un virtualbox-3.1 <none> <none> (no description available)
un virtualbox-3.2 <none> <none> (no description available)
un virtualbox-4.0 <none> <none> (no description available)
un virtualbox-4.1 <none> <none> (no description available)
un virtualbox-4.2 <none> <none> (no description available)
un virtualbox-4.3 <none> <none> (no description available)
un virtualbox-5.0 <none> <none> (no description available)
un virtualbox-5.1 <none> <none> (no description available)
un virtualbox-5.2 <none> <none> (no description available)
un virtualbox-6.0 <none> <none> (no description available)
un virtualbox-6.1 <none> <none> (no description available)
ii virtualbox-dkms 6.1.18-dfsg-5 amd64 x86 virtualization solution - kernel modul>
un virtualbox-guest-additions-iso <none> <none> (no description available)
un virtualbox-guest-dkms <none> <none> (no description available)
un virtualbox-guest-modules <none> <none> (no description available)
un virtualbox-modules <none> <none> (no description available)
ii virtualbox-qt 6.1.18-dfsg-5 amd64 x86 virtualization solution - Qt based use>
un virtualbox-source <none> <none> (no description available)
user@icy:~$ ls -al /usr/src/
total 24
drwxr-xr-x 6 root root 4096 Jun 13 10:51 .
drwxr-xr-x 14 root root 4096 Apr 23 2020 ..
drwxr-xr-x 3 root root 4096 Mai 14 2020 libdvd-pkg
drwxr-xr-x 24 root root 4096 Jun 5 13:55 linux-headers-5.11.0-18
drwxr-xr-x 7 root root 4096 Jun 5 14:13 linux-headers-5.11.0-18-generic
drwxr-xr-x 11 root root 4096 Jun 13 10:51 virtualbox-6.1.18
user@icy:~$ ls -al /lib/modules/$(uname -r)/updates/dkms
total 680
drwxr-xr-x 2 root root 4096 Jun 13 10:52 .
drwxr-xr-x 3 root root 4096 Jun 13 10:52 ..
-rw-r--r-- 1 root root 620706 Jun 13 10:52 vboxdrv.ko
-rw-r--r-- 1 root root 17474 Jun 13 10:52 vboxnetadp.ko
-rw-r--r-- 1 root root 44378 Jun 13 10:52 vboxnetflt.ko
user@icy:~$ dkms status
virtualbox, 6.1.18, 5.11.0-18-generic, x86_64: installed
user@icy:~$ grep -i virtualbox /etc/apt/sources.list
<No output here>
user@icy:~$ grep -i virtualbox /etc/apt/sources.list.d/*.list
<No output here>
I had the same problem after upgrading from 20.10 to 21.04 and managed to fix it without a complete reinstall by uninstalling everything related to module-builds:
- I removed all versions of build-essential, gcc, clang, binutils, libtool, autoconf, flex, and bison.
- I removed all packages named linux-headers-* and removed the corresponding directories in /usr/src.
- I removed and purged the dkms package and manually removed remaining files in /var/lib/dkms.
Then I reinstalled flex and bison (needed by linux-headers, but not installed automatically) and dkms.