Bad behaviour of native win32 application when replacing the exe with some users using it

Short answer: DON'T CHANGE FILES THAT ARE CURRENTLY IN USE OR RUNNING IN ACTIVE MEMORY.

If the file is open and in use, you're going to have problems. Pulling a file out from under the operating system/application while there are open file handles is a big no-no. I'm surprised it would even let you rename the file.

It's not necessarily caching something, you have active applications running that don't assume the files will be changing without notification.

This is why often Windows forces you to restart for certain updates, many updaters for software force you to close your browser or other "related" programs (Java comes to mind) to finish updating, or other updaters "restart" the program they're updating (like Adobe installers).