systemd-udevd causes high load after upgrade to Ubuntu 21.10 server on Raspberry PI 4

After upgrading my Raspberry Pi 4 8GB from Ubuntu 21.04 to 21.10 I got high load due to systemd-udevd. I made a clean install of Ubuntu server 21.10 but the problems remain. I use a SSD (no SD card). When I boot it looks fine, but after about a day the load is above 25 and most of it seems to be due to >15 systemd-udevd processes. I run a small installation of MicroK8s and a MySQL server and I think the load is not related to those. Before the upgrade the load was stable around 4.

Here is an example of the top command printouts:

2253733 mysql     20   0 2792212 436376  32344 S  71.0   5.5 661:02.75 mysqld
2218900 root      20   0    9100    716      0 R  66.8   0.0 920:43.38 apiservice-kick
 780593 root      20   0 3545776 775276 104908 S  63.2   9.7   1076:01 kubelite
 802755 root      20   0  761976  46732  34148 S  13.0   0.6   0:00.40 calico-node
 910416 root      20   0   23240   2948      0 R  12.7   0.0 193:44.68 systemd-udevd
 926403 root      20   0   23240   2948      0 R  10.7   0.0 192:11.89 systemd-udevd
 746105 root      20   0  265968  39976   9208 S  10.1   0.5   2:19.20 flask
 411700 root      20   0   31172  10920      0 R   9.8   0.1 287:25.06 systemd-udevd
1698353 root      20   0   23240   2948      0 R   8.8   0.0 142:39.40 systemd-udevd
    958 root      20   0  145596 126728   3708 R   8.1   1.6  82:53.44 systemd-udevd
1141818 root      20   0   23240   2948      0 R   7.5   0.0 175:48.13 systemd-udevd
 755129 root      20   0   22708   2488      0 R   7.2   0.0 214:38.38 systemd-udevd
2919424 root      20   0   23240   2936      0 R   6.5   0.0  85:49.04 systemd-udevd
1258055 root      20   0   23240   2948      0 R   6.2   0.0 168:29.82 systemd-udevd
1383762 root      20   0   23240   2948      0 R   5.5   0.0 160:01.94 systemd-udevd
 599455 root      20   0   35532  15252      0 R   5.2   0.2 246:01.83 systemd-udevd
 867551 root      20   0   23240   2948      0 R   5.2   0.0 197:39.14 systemd-udevd
2084650 root      20   0   23240   2948      0 R   5.2   0.0 123:10.51 systemd-udevd
1906684 root      20   0   23240   2948      0 R   4.9   0.0 131:04.74 systemd-udevd
 770329 root      20   0   22708   2488      0 R   4.6   0.0 212:20.44 systemd-udevd
 890295 root      20   0   23240   2948      0 R   4.2   0.0 195:47.40 systemd-udevd
 780768 root      20   0 1932764  96716  27984 S   2.3   1.2  99:27.87 containerd

When I run udevadm monitor I get repeated sequences like this:

KERNEL[124229.147890] add      /devices/virtual/net/calico_tmp_B (net)
KERNEL[124229.148029] add      /devices/virtual/net/calico_tmp_B/queues/rx-0 (queues)
KERNEL[124229.148101] add      /devices/virtual/net/calico_tmp_B/queues/tx-0 (queues)
KERNEL[124229.148660] add      /devices/virtual/net/calico_tmp_A (net)
KERNEL[124229.148764] add      /devices/virtual/net/calico_tmp_A/queues/rx-0 (queues)
KERNEL[124229.148830] add      /devices/virtual/net/calico_tmp_A/queues/tx-0 (queues)
UDEV  [124229.313680] remove   /devices/virtual/net/calico_tmp_A/queues/rx-0 (queues)
KERNEL[124229.408421] remove   /devices/virtual/net/calico_tmp_A/queues/rx-0 (queues)
KERNEL[124229.408530] remove   /devices/virtual/net/calico_tmp_A/queues/tx-0 (queues)
KERNEL[124229.408614] remove   /devices/virtual/net/calico_tmp_A (net)
KERNEL[124229.515488] remove   /devices/virtual/net/calico_tmp_B/queues/rx-0 (queues)
KERNEL[124229.515593] remove   /devices/virtual/net/calico_tmp_B/queues/tx-0 (queues)
KERNEL[124229.515675] remove   /devices/virtual/net/calico_tmp_B (net)
UDEV  [124229.727966] remove   /devices/virtual/net/calico_tmp_A/queues/tx-0 (queues)
UDEV  [124230.646980] remove   /devices/virtual/net/calico_tmp_A (net)
KERNEL[124238.148409] add      /devices/virtual/net/calico_tmp_B (net)
KERNEL[124238.148522] add      /devices/virtual/net/calico_tmp_B/queues/rx-0 (queues)
KERNEL[124238.148587] add      /devices/virtual/net/calico_tmp_B/queues/tx-0 (queues)
KERNEL[124238.149038] add      /devices/virtual/net/calico_tmp_A (net)
KERNEL[124238.149132] add      /devices/virtual/net/calico_tmp_A/queues/rx-0 (queues)
KERNEL[124238.149196] add      /devices/virtual/net/calico_tmp_A/queues/tx-0 (queues)
KERNEL[124238.257022] remove   /devices/virtual/net/calico_tmp_A/queues/rx-0 (queues)
KERNEL[124238.257124] remove   /devices/virtual/net/calico_tmp_A/queues/tx-0 (queues)
KERNEL[124238.257209] remove   /devices/virtual/net/calico_tmp_A (net)
KERNEL[124238.263626] remove   /devices/virtual/net/calico_tmp_B/queues/rx-0 (queues)
KERNEL[124238.263727] remove   /devices/virtual/net/calico_tmp_B/queues/tx-0 (queues)
KERNEL[124238.263808] remove   /devices/virtual/net/calico_tmp_B (net)
UDEV  [124238.536622] add      /devices/virtual/net/calico_tmp_A (net)
UDEV  [124238.939643] add      /devices/virtual/net/calico_tmp_A/queues/rx-0 (queues)
UDEV  [124239.613507] add      /devices/virtual/net/calico_tmp_A/queues/tx-0 (queues)

If I reboot, it starts OK, but then the load builds up again.

Any help is appreciated.


Solution 1:

I found the solution. On the MicroK8s Troubleshooting page it mentions that additional modules should be installed for Ubuntu 21.10 on Raspberry Pi. After running sudo apt install linux-modules-extra-raspi the load is down on 1.3 and the problem is gone.