What game was being played on The Office?

This is a fake game. A demo created specifically for the show made to look similar to real games.

Watch here (thanks @BlueRaja for providing the video link), shown between 1:30 and 2:30

Reasons:

1. The game plays itself. In the scene, we see the character with hands on and off the mouse, the player continues to move, even when nobody has their hands on the controls. Additionally, the character never uses the keyboard, only ever the mouse. It's quite hard to play a FPS with just a mouse. This is a clear indication, that this is a scripted demo running and the character just pretends to play.

2. The game is very simple. The animation consists of moving forward and occasionally an explosion. This doesn't feel like a real game, but a simple tech demo.

3. The technology is mismatched. This episode aired in March 2010. The monitor looks like a standard TFT screen from roughly 2005, while the game looks like a mid 90s FPS. While it's certainly possible to play such a game on a modern machine, it seems strange for the producers to show such an old game and getting it to run on a Windows Vista or newer machine isn't all that easy. DOSBox would likely be required and I don't see the show producers make that effort for a background animation.

4. A lot of elements look very similar to Doom (character portrait in the UI, the weapon model).

5. The games UI is at the top of the screen, which is very unusual. The most important UI elements in FPS are always at the bottom, because that's where the weapon model is. The only game with top UI elements is Rise of the Triad and that only displayed less important information (e.g. score) on the top.

6. The player character is named (Mad dog), this is again taken from Rise of the Triad and serves no purpose, no other game I know did that.

7. Searching Mad Dog FPS on google only yields an FMV western game. The character is unknown otherwise.

All of this leads me to the conclusion, that this is an animation made to look like a 90s FPS specifically for the show. Why do this? When you show a real game, you run into licensing trouble. The game must be credited and you need permission from the owners to show it. Especially, when the game is just an unimportant background element, the games owner has little incentive to consider it advertisement. Thus it's simpler and cheaper to just create a short animation. This also means, the actor doesn't have to learn (at least rudimentary) how to play the game, they just need to pretend to play it.