Overlaying on a 3D fullscreen application

I want to display some custom graphics on top of a 3rd party fullscreen Windows application.

Have you played any Steam games? It has an executable, GameOverlayUI.exe that lets you access Steam windows from within a game. (The GUI elements look custom-drawn, I don't know if that is a necessity; but they look exactly the same inside a game as they do on the desktop.) It even goes as far to 'dim' the game graphics in the background.

I'm wondering how one could go about writing such an application.

I'm also wondering how broad the scope of solutions would be. Could you use one method for all OpenGL applications? For all Direct3D minus DirectDraw applications? For all fullscreen Windows applications?

I'm looking for more 'modern' and generic solutions - overlaying a command prompt or a 320x240 indexed color application isn't a concern.

Edit: Some clarification: The fullscreen application isn't mine. It can be anything. Most probably, it will be a 3D game. I want to display, say a digital clock in the bottom right corner of the screen, on top of the game graphics. I would like to avoid tricks such as injecting code or debugging the process, if possible.

Solution 1:

Well, here's another example. Xfire is a chat program that overlays its interface into games so you can receive messages while playing. The way they do this is by editing the d3d/opengl DLL at the process memory level, like injecting assembly jumps. I know this because their method was causing my mod to crash, and I actually saw the strange assembly code they were injecting into the d3d9.dll.

So here's how Xfire does this:

1. target the process of the game
2. search its process memory for d3d.dll/opengl.dll
3. inject into the process a DLL containing the custom interface logic
4. connect interface's functions to the program flow by manually writing assembly jumps in the d3d/opengl functions found in the process memory

There's no other conceivable way since you need to hijack the graphics device that belongs to that game. I'm willing to bet that Steam does it the same way as Xfire. So yeah, no easy way to do this unless someone created a helpful library to do everything you need to do at the low level.

You also asked about dimming the game graphics under your overlay. This is just a simple DrawPrimitive call to draw a filled transparent rectangle over the screen.

Solution 2:

Created a Gist with full code here

This is a pretty esoteric art, and there are several ways to do this. I prefer what's called a Direct3D Wrapper. It's been years since I've done this, but I did it with a game once. I may have left out some details, but I just hope to illustrate the idea.

All Direct3D9 games need to call IDirect3DDevice9::EndScene after the screen is rendered and ready to be drawn. So in theory, we can put custom code inside EndScene that draws what we want over the game. We do this by creating a class that looks like IDirect3DDevice9 to the game, but it's really just a wrapper that forwards all function calls to the real class. So now, in our custom class, our wrapper function for EndScene looks like this:

HRESULT IDirect3DDevice9::EndScene()
{
   // draw overlays here

   realDevice.EndScene();
}

Then we compile it into a DLL called d3d9.dll, and drop it in the game executable's directory. The real d3d9.dll should be in the /system32 folder, but the game first looks in the current directory, and instead loads ours. Once the game loads, it uses our DLL to create an instance of our wrapper class. And now each time EndScene is called, our code is executed to draw what we want to the screen.

I think you can try the same principle with OpenGL. But to prevent tampering, I think some modern games try to prevent this.