How and why does QuickEdit mode in Command Prompt freeze applications?
I recently ran into an issue with Command Prompt on Windows where QuickEdit mode was enabled and clicking the window was selecting text and hanging a running program. This is, apparently, known behaviour—I found a few questions related to it:
- Command Line Windows Hanging in RDP Windows
- Windows Console Application Getting Stuck
- How to disable QuickEdit Mode for individual scripts
How is the application "paused"/"suspended"? Is the process similar to the SIGSTOP signal on *nix? (I am also interested in understanding why this functionality exists in the first place? It seems unintuitive and dangerous.)
Solution 1:
This is very much by design. There's no reasonable way a user can select text when your program keeps scrolling the content of the console window. So the console host program simply stops reading your stdout/stderr output and your program hangs until the user completes the operation. This can be changed, you'll have to call Get+SetConsoleMode() and turn off the ENABLE_QUICK_EDIT_MODE option.
Do note that this "hang" isn't fundamentally different from the execution pauses you get when your program generates stdout output at a rate far higher than the console host can consume it. Albeit that those delays are finite.
And it isn't the only way the user can stop your program, they can also simply press Ctrl+S. Pressing Ctrl+Q resumes it again. If you're old enough then you might recognize these control codes as Xon/Xoff, handshake characters for a terminal. Which is what a console really is, a simple emulation of a terminal the way they were used back in the 1970s. This can be changed too, you'll have to stop relying on the built-in buffered console input and switch to ReadConsole(). Or by turning off the ENABLE_LINE_INPUT console option, not so sure what side-effects that has since you didn't mention any language runtime, you'll have to try.
And of course terminating your program is very easy. You get EOF on stdin when the user types Ctrl+Z, that ought to end your program. And there is Ctrl+C and Ctrl+Break for an instant termination, regardless what your program is doing. You can get a notification for these with SetConsoleCtrlHandler() but you can't block it.
If the default behavior is dangerous and risk the health of a human then I'd strongly suggest you hire a consultant. And won't know who wrote this answer.
Solution 2:
To answer the how I was recently debugging the same problem in a python script and captured a stack trace using windbg.
The call to WriteConsole eventually leads to the NtDeviceIoControlFile syscall and the kernel does not return until a keypress happens or the QuickEdit mode is changed. So if you never write to the console, your process is safe from QuickEdit mode freeze. And your user will never need to copy anything. So wait what is QuickEdit mode for again?