How to remotely detect windows has completed patch configuration after reboot

We're planning to automate the creation of VMs for our build infrastructure so that we can:

1. Scale the build resources based on demand, e.g. by adding more build agents when required and removing them when not required
2. Recreate all or part of the build environment if / when machines die
3. Duplicate the build environment when we need a test set up

One of the steps in this process is automating the creation of the VM base images (in our case using Hyper-V). For that we have a script that:

1. Creates new VHDX from ISO with the Convert-WindowsImage script. We're currently using Windows 2012R2 but will be looking to get started with 2016 as soon as it is available.
2. Adds an unattend script to the new VHDX with all the base configuration we need
3. Updates the VHDX with the latest windows patches using the Apply-WindowsUpdate script
4. Creates a new Hyper-V VM based on the VHDX and starts it
5. Waits for the VM to boot and wait for the WinRM service to be ready to accept remote connections
6. Waits for windows to complete the initial configuration and the configuration of the new patches
7. Applies any further patches
8. Reboots to complete the configuration of the latest patches
9. Waits for windows to complete configuring the patches
10. Pushes a sysprep script to machine and invokes that script. This runs sysprep and then turns the machine off
11. Deletes the VM but keeps the VHDX
12. Removes sysprep and unattend files from the VHDX and then compacts the VHDX
13. Moves VHDX to template location and mark as read-only

The problem we're experiencing is in steps 6 and 9. Ideally we wait for all the configuration to be complete before we reboot / shut down the machine but there does not seem to be a way to detect that windows has finished the configuration stage.

When going through the UI it is very clear when either step is done because the log-in UI doesn't show up until the process is ready. However when using WinRM to remotely connect to the machine this is less clear because WinRM provides access to the machine before it is done with the configuration work.

So the question is what is the most fool proof way to detect over a remote connection that Windows has finished configuring updates etc. so that we can reboot / shut down the machine without causing issues later on.

------ EDIT -----

In the end we're using a modified version of Katherine's answer in that our script also waits for windeploy and ngen to complete. Given that ngen doesn't complete until well after the OS has finished initializing that works, and as a bonus the final VHDX will have all of the .NET framework ngen-ed which means we don't have to deal with that when we create new VMs of the template disk. Both script that we use to create the VHDX template and the scripts to create the local test environment are on github in case anybody is interested.

Solution 1:

This might sound like kind of a weird answer, but...

There's a PowerShell script for checking to see if there are available updates for Nagios. You could probably use this script or a variant for your purposes, without Nagios.

As for whether they're in progress, check for whether or not Wuauclt and TrustedInstaller are running. Microsoft's advice about updates on Server Core might help here:

Depending on the updates that are installed, you might need to restart the computer, although the system will not notify you of this. To determine if the installation process has completed, use Task Manager to verify that the Wuauclt or Trusted Installer processes are not actively running. You can also use the methods in the “Viewing installed updates” section to check the list of installed updates.

You can probably pull that information with something like Get-Process -Computername YourImage TrustedInstaller.exe. After both the Wuauclt and TrustedInstaller processes have finished, it should be safe to reboot.

Solution 2:

Each Windows update patch will write several events in the Setup event log.

• Event ID 1 - Initiating changes for package KB####
• Event ID 4 - A reboot is necessary before package KB#### can be changed to installed state
• Event ID 2 - Package KB#### was successfully changed to the Installed state

One way to determine all patches had been applied would be to loop a check on Event ID 4. Compare the time of that event to the current time. If no event ID 4's had been written for 5 or 10 minutes, then all pataches are probably done, and ready to reboot.

I'm not clear if you want to do the first reboot when patches are done installing (event4), or the second reboot after they are done configuring (event 2). This code does the former. Simply change the filterHashTable to event id 2 for the other reboot before your step 10.

$target = "bart"
$found = $false
while (-not $found) {
    $lastEvent4 = (get-winevent -comp $target -maxEvents 1 -filterHashTable @{ Logname='Setup'; id = '4';}).timeCreated
    if (((get-date) - $lastEvent4).totalMinutes -gt 10) {
        "do reboot"
        restart-computer -comp -$target
        $found = $true
    } else {
        "wait"
        start-sleep 60
    }
}