systemd-udevd causes high load after upgrade to Ubuntu 21.10 server on Raspberry PI 4
After upgrading my Raspberry Pi 4 8GB from Ubuntu 21.04 to 21.10 I got high load due to systemd-udevd. I made a clean install of Ubuntu server 21.10 but the problems remain. I use a SSD (no SD card). When I boot it looks fine, but after about a day the load is above 25 and most of it seems to be due to >15 systemd-udevd processes. I run a small installation of MicroK8s and a MySQL server and I think the load is not related to those. Before the upgrade the load was stable around 4.
Here is an example of the top command printouts:
2253733 mysql 20 0 2792212 436376 32344 S 71.0 5.5 661:02.75 mysqld
2218900 root 20 0 9100 716 0 R 66.8 0.0 920:43.38 apiservice-kick
780593 root 20 0 3545776 775276 104908 S 63.2 9.7 1076:01 kubelite
802755 root 20 0 761976 46732 34148 S 13.0 0.6 0:00.40 calico-node
910416 root 20 0 23240 2948 0 R 12.7 0.0 193:44.68 systemd-udevd
926403 root 20 0 23240 2948 0 R 10.7 0.0 192:11.89 systemd-udevd
746105 root 20 0 265968 39976 9208 S 10.1 0.5 2:19.20 flask
411700 root 20 0 31172 10920 0 R 9.8 0.1 287:25.06 systemd-udevd
1698353 root 20 0 23240 2948 0 R 8.8 0.0 142:39.40 systemd-udevd
958 root 20 0 145596 126728 3708 R 8.1 1.6 82:53.44 systemd-udevd
1141818 root 20 0 23240 2948 0 R 7.5 0.0 175:48.13 systemd-udevd
755129 root 20 0 22708 2488 0 R 7.2 0.0 214:38.38 systemd-udevd
2919424 root 20 0 23240 2936 0 R 6.5 0.0 85:49.04 systemd-udevd
1258055 root 20 0 23240 2948 0 R 6.2 0.0 168:29.82 systemd-udevd
1383762 root 20 0 23240 2948 0 R 5.5 0.0 160:01.94 systemd-udevd
599455 root 20 0 35532 15252 0 R 5.2 0.2 246:01.83 systemd-udevd
867551 root 20 0 23240 2948 0 R 5.2 0.0 197:39.14 systemd-udevd
2084650 root 20 0 23240 2948 0 R 5.2 0.0 123:10.51 systemd-udevd
1906684 root 20 0 23240 2948 0 R 4.9 0.0 131:04.74 systemd-udevd
770329 root 20 0 22708 2488 0 R 4.6 0.0 212:20.44 systemd-udevd
890295 root 20 0 23240 2948 0 R 4.2 0.0 195:47.40 systemd-udevd
780768 root 20 0 1932764 96716 27984 S 2.3 1.2 99:27.87 containerd
When I run udevadm monitor I get repeated sequences like this:
KERNEL[124229.147890] add /devices/virtual/net/calico_tmp_B (net)
KERNEL[124229.148029] add /devices/virtual/net/calico_tmp_B/queues/rx-0 (queues)
KERNEL[124229.148101] add /devices/virtual/net/calico_tmp_B/queues/tx-0 (queues)
KERNEL[124229.148660] add /devices/virtual/net/calico_tmp_A (net)
KERNEL[124229.148764] add /devices/virtual/net/calico_tmp_A/queues/rx-0 (queues)
KERNEL[124229.148830] add /devices/virtual/net/calico_tmp_A/queues/tx-0 (queues)
UDEV [124229.313680] remove /devices/virtual/net/calico_tmp_A/queues/rx-0 (queues)
KERNEL[124229.408421] remove /devices/virtual/net/calico_tmp_A/queues/rx-0 (queues)
KERNEL[124229.408530] remove /devices/virtual/net/calico_tmp_A/queues/tx-0 (queues)
KERNEL[124229.408614] remove /devices/virtual/net/calico_tmp_A (net)
KERNEL[124229.515488] remove /devices/virtual/net/calico_tmp_B/queues/rx-0 (queues)
KERNEL[124229.515593] remove /devices/virtual/net/calico_tmp_B/queues/tx-0 (queues)
KERNEL[124229.515675] remove /devices/virtual/net/calico_tmp_B (net)
UDEV [124229.727966] remove /devices/virtual/net/calico_tmp_A/queues/tx-0 (queues)
UDEV [124230.646980] remove /devices/virtual/net/calico_tmp_A (net)
KERNEL[124238.148409] add /devices/virtual/net/calico_tmp_B (net)
KERNEL[124238.148522] add /devices/virtual/net/calico_tmp_B/queues/rx-0 (queues)
KERNEL[124238.148587] add /devices/virtual/net/calico_tmp_B/queues/tx-0 (queues)
KERNEL[124238.149038] add /devices/virtual/net/calico_tmp_A (net)
KERNEL[124238.149132] add /devices/virtual/net/calico_tmp_A/queues/rx-0 (queues)
KERNEL[124238.149196] add /devices/virtual/net/calico_tmp_A/queues/tx-0 (queues)
KERNEL[124238.257022] remove /devices/virtual/net/calico_tmp_A/queues/rx-0 (queues)
KERNEL[124238.257124] remove /devices/virtual/net/calico_tmp_A/queues/tx-0 (queues)
KERNEL[124238.257209] remove /devices/virtual/net/calico_tmp_A (net)
KERNEL[124238.263626] remove /devices/virtual/net/calico_tmp_B/queues/rx-0 (queues)
KERNEL[124238.263727] remove /devices/virtual/net/calico_tmp_B/queues/tx-0 (queues)
KERNEL[124238.263808] remove /devices/virtual/net/calico_tmp_B (net)
UDEV [124238.536622] add /devices/virtual/net/calico_tmp_A (net)
UDEV [124238.939643] add /devices/virtual/net/calico_tmp_A/queues/rx-0 (queues)
UDEV [124239.613507] add /devices/virtual/net/calico_tmp_A/queues/tx-0 (queues)
If I reboot, it starts OK, but then the load builds up again.
Any help is appreciated.
Solution 1:
I found the solution. On the MicroK8s Troubleshooting page it mentions that additional modules should be installed for Ubuntu 21.10 on Raspberry Pi. After running sudo apt install linux-modules-extra-raspi the load is down on 1.3 and the problem is gone.