How to prevent an application from running in fullscreen mode in Windows 7?

Alt + Enter often works (mostly on DirectX applications). This seems to be a tradition more than anything else, but I've found a surprisingly large number of programs support it. Also, Esc is worth a shot.

There is also a program called DxWnd that is designed specifically for this purpose: forcing fullscreen programs into windowed mode. I haven't used it in a long time, but it's still actively developed and it's worth a shot. It was originally designed for and works best with DirectX, but it seems OpenGL (another graphics library similar to DirectX) and GDI (general Windows programs often use this) support has been added. See here for download links.

Although it's not the greatest solution, running the application from within a virtual machine accomplishes the desired effect. The screen resolution of the virtual machine can be set as desired, and then the application fills only the virtual machine's window space.