apt/unattended-upgrades stalls shutdown

When unattended-upgrades is installed, 9 out of 10 shutdowns/reboots hang while "starting unattended upgrades shutdown". This hang stalls the shutdown process for 5-10 mins.

If I disable unnattended-upgrades via the /etc/apt/apt.conf.d/20auto-upgrades and/or 50unattended-upgrades, the problems occurs.

If I terminate the service before shutdown/reboot (sudo service unattended-upgrades stop) the problem still occurs.

If I remove the package (sudo apt remove unattended-upgrades) the problem no longer occurs.

This occurs on a freshly installed version of Ubuntu Server 16.04.1 (both unattended-upgrades installed via install GUI or manual install of unattended-upgrades)

Both Kern.log & syslog do not show the shutdown process (I believe because the filesystems have already unmounted)

Has anyone else seen or fixed this issue? Going crazy trying to troubleshoot it.

Looking around to get closer to the root cause

The problem seems to be the script running at shutdown.

I identified the corresponding file with:

find /etc/systemd -name *unattended*

which gaves me the related systemd script:

/etc/systemd/system/shutdown.target.wants/unattended-upgrades.service

which then told me the script executed on shutdown:

/usr/share/unattended-upgrades/unattended-upgrade-shutdown

Investigating deeper to find the root cause

within this script there is a section in line 120 related to the section in /etc/apt/apt.conf.d/50unattended-upgrades -> Unattended-Upgrade::InstallOnShutdown

Line 120 of /usr/share/unattended-upgrades/unattended-upgrade-shutdown:

if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", False):

The problem: it expects the keyword "False" while in the apt conf we should add "false" (exact string comparison)!

Solution

I was able to fix/workaround the stalling shutdown in 3 different ways:

Workaround A

write "False" instead of "false" in /etc/apt/apt.conf.d/50unattended-upgrades

This setting is upgrade safe until a real fix is provided because the file we change here gets not overwritten by an update of unattended-upgrades. Problem: When the root cause gets fixed this will result in a stalling shutdown again so I suggest to combine this with Workaround B.

OR: Workaround B

decrease the wait time in /etc/systemd/system/shutdown.target.wants/unattended-upgrades.service from default to 15 seconds:

vim /etc/systemd/system/shutdown.target.wants/unattended-upgrades.service

[Service]
Type=oneshot
ExecStart=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
TimeoutStartSec=15

This setting is NOT upgrade safe because the file we change here may get overwritten by an update of unattended-upgrades. Besides this it is really far away from fixing something but it will ensure that your system will not wait several minutes when shutting down. Keep in mind that after an upgrade of unattended-upgrades you may have to set this again!

OR: Fix C (have to be reported upstream)

fix /usr/share/unattended-upgrades/unattended-upgrades-shutdown to expect "false" instead of "False"

patching /usr/share/unattended-upgrades/unattended-upgrade-shutdown:

--- /tmp/unattended-upgrade-shutdown    2017-02-03 14:53:03.238103238 +0100
+++ /tmp/unattended-upgrade-shutdown_fix    2017-02-03 14:53:17.685589001 +0100
@@ -117,7 +117,7 @@
     # run it
     p = None
     apt_pkg.init_config()
-    if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", False):
+    if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", false):
         env = copy.copy(os.environ)
         env["UNATTENDED_UPGRADES_FORCE_INSTALL_ON_SHUTDOWN"] = "1"
         logging.debug("starting unattended-upgrades in shutdown mode")

Conclusion

tbh only the last one is a real fix. the both other options are just workarounds until the real fix would be implemented.

This has to be done upstream and as this affects both Debian (tested on Debian Stretch) and Ubuntu (tested on Ubuntu 16.04.1) for both distributions.

I have opened a bug report here: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1661611

This has been fixed in systemd

https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600

(original report: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1661611)