How to set “Override high DPI scaling” to “Application” programmatically through code
I was able to save the following into a .bat file in order to set DPI scaling.
This command sets HIGHDPI scaling:
REG ADD "HKCU\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers" /V "C:\Program Files (x86)\VidSys\VidSys Out-of-Process Viewer Framework\External Process\NvrViewerExternalProcess.exe" /T REG_SZ /D ~HIGHDPIAWARE /F
Obviously, you need to replace the path in the command with the absolute path to your EXE.
To set other types of scaling, use the EXE's property page to set the type of scaling that you want, then check the Windows registry for the key named in the above command. Then adjust the command by substituting in the string you find in the registry.