Background

• Our servers are Server 2016 Version 1607
• Our servers have been created at varying times, some have been up for a few months, others a few years
• We deploy our patches via SCCM

The Issue

During our maintenance windows when we apply the latest and greatest patches to our Windows Server 2016 VMs, we noticed that occasionally, (more often as time goes on), Cumulative Update (CU) patches end up either stuck, or taking hours to apply. We believe that the reason for these long times are the presence and amount of superseded updates on the server.

So that leaves the question - why are these superseded updates accumulating and not being cleaned up? And how can we clean them up?

TL:DR

In short, the superseded updates were being caused by a Windows Owned scheduled task failing. This scheduled task is responsible for cleaning up superseded updates. To fix this, you're able to run a Dism.exe command to manually cleanup these superseded updates. This allows for faster, more reliable patching during maintenance windows. Find more information about these scheduled tasks and Dism.exe commands here.

Superseded Updates In The Event Log

So firstly lets talk about how we know there are CUs and other updates that are being cleaned up and delaying patch installation. For people new to investigating patching on Windows Server, there is a Windows Event Viewer log called Setup which has events related to patch installation. In this event viewer log, we can see an event denoting that a KB was marked as superseded and is to be removed. Below is what such an event looks like.

These events would always occur right as we are starting patching during our maintenance window. This means that much of our time spent patching is actually spent handling these superseded updates, causing our patches to take MUCH LONGER than we planned for.

So why are these superseded updates lingering and only being cleaned up during patching?

Windows Task Scheduler StartComponentCleanup

While researching the issue, I came across an article detailing Windows built-in capability to clean up out-of-date components. It turns out that this is exactly what I was looking for. The cleanup that this article discusses is cleaning up old components, this correlates to the removal of the superseded updates.

Clean Up the WinSxS Folder - Task Scheduler

This article gives information about a built-in Task Scheduler task: Library\Microsoft\Windows\Servicing\StartComponentCleanup

The StartComponentCleanup task was created in Windows 8 to regularly clean up components automatically when the system is not in use. This task is set to run automatically when triggered by the operating system. When run automatically, the task will wait at least 30 days after an updated component has been installed before uninstalling the previous versions of the component. If you choose to run this task, the task will have a 1 hour timeout and may not completely clean up all files.

Upon reviewing this task on our servers, I found many instances of this task either failing to run or being stopped before it was able to complete. This I believe was caused due to the time limitations discussed in the excerpt from Microsoft above, and I believe that this task failing on many of our servers is the reason we are experiencing surplus of superseded updates. When this task can't run, superseded updates can't be cleaned up on a regular basis.

Dism.exe

The article previously mentioned then goes on to discuss Dism.exe commands that can be used to perform this manual cleanup.

Clean Up the WinSxS Folder - Dism.exe

Effectively, you can run the same command that the Scheduled Task was running to clean up out-of-date components, albeit, without the limitations imposed by the Scheduled Task.

I found it beneficial to first run the Dism.exe command with the AnalyzeComponentStore flag. This returns whether or not there are any out-of-date components and whether or not you should run the cleanup command.

Dism.exe /online /Cleanup-Image /AnalyzeComponentStore

C:\Users\StackOverflow>"%SystemRoot%\System32\Dism.exe" /online /Cleanup-Image /AnalyzeComponentStore
Deployment Image Servicing and Management tool
Version: 10.0.14393.4169
Image Version: 10.0.14393.4169
[==========================100.0%==========================]
Component Store (WinSxS) information:
Windows Explorer Reported Size of Component Store : 9.52 GB
Actual Size of Component Store : 9.12 GB
    Shared with Windows : 5.29 GB
    Backups and Disabled Features : 3.15 GB
    Cache and Temporary Data : 672.30 MB
Date of Last Cleanup : 2019-04-10 08:58:06
Number of Reclaimable Packages : 3
Component Store Cleanup Recommended : Yes
The operation completed successfully.

If you see this in your result: Component Store Cleanup Recommended : Yes then you can likely proceed with a component cleanup using the following command. Note that this can take quite some time to run.

Dism.exe /online /Cleanup-Image /StartComponentCleanup

And running this command did exactly what I was looking for! Running this command kicked off the process of Windows removing the superseded updates (as seen by the screenshot below). This is important because it gives us the ability to proactively remove superseded updates prior to maintenance windows. This allows our maintenance windows to be more predictable, with less chance of exceeding the window or getting stuck.

Conclusion

In short, the superseded updates were being caused by a Windows Owned scheduled task failing. This scheduled task is responsible for cleaning up superseded updates. To fix this, you're able to run a Dism.exe command to manually cleanup these superseded updates. This allows for faster, more reliable patching during maintenance windows.

Bonus Info Regarding Patching Windows Server Version 1607

Below is an article talking about the mechanisms used by Microsoft to patch the operating system. It gives information related to how and why the 1607 version of Windows takes longer to patch than others. If your on this version, it may be worth a look.

How to improve Windows cumulative update installation times