What tells the Windows installer service that another install is in progress?
The Windows Installer runtime acquires a global mutex with the name _MSIExecute during certain stages of an (un)installation. A mutex can only be owned by one thread at a time; for a global mutex, that is one thread on the entire system. Any subsequent attempts to obtain ownership of the mutex will fail until the current owner releases it.
A mutex is a transient object in the Windows kernel. It is not a file on disk at any point. A mutex is automatically released if the current owning thread exits without explicitly releasing it.
This is the same mechanism some programs use to prevent multiple instances of the program from running.
To answer the other implied question of which process is holding that mutex, you can use tools such as Microsoft/Sysinternals' Process Explorer or Handle to find mutexes. In Process Explorer, it should come up in the handle search (Ctrl+F).
Once you've found the process, you'll need to decide what to do with it. The safest option is usually to wait, though if it has somehow gotten stuck in an (un)installation step, you might have no choice but to kill it. Of course, killing it could have consequences, e.g. leaving a half-(un)installed program lying around - MSI is supposed to guarantee rollbacks, but those are difficult when the process performing them is killed! It's also difficult to judge whether it's actually stuck, as some steps can take a very long time to complete.
See also:
- https://docs.microsoft.com/en-us/windows/win32/msi/-msiexecute-mutex
- https://stackoverflow.com/questions/32049193/how-to-prevent-msi-error-another-program-is-being-installed