How do I dismiss alert popups on an unattended windows server?

Solution 1:

The software is flawed. Contact the vendor and tell them that server software has no place throwing GUI errors. Software designed to run as a server service must have NO GUI interaction in the server process and should run without need a logged in session using the built-in Windows service API.

Solution 2:

It is true that server-class software is not supposed to do stuff like pop up GUI dialog boxes, and that you should be using software that was meant for servers, and you should chide the vendor and make them feel bad...

...but you might want to be able to do something about it right now.

You need some automatable way to detect that the application or service is in this state. Once you've found that, you can use something like WASP for Powershell to automate mouse clicks, or even just Get-Process "GhettoProgram" | Stop-Process and then restart it.