What is the optimal position of a sentry gun?

The best place to locate a sentry is where the enemy doesn't know it is. However, that only works once at most if you're lucky and they didn't hear it for some reason.

When strategizing with your sentry placement, remember to rely on yourself and your team to back it up. SGs are phenomenally accurate and can mow down most classes in 1-2 seconds. One of the most effective locations to place sentries is where enemies coming around a corner will be unable to see and hit it before they are well in your team's field of fire. Most people will get caught up on shooting whoever's shooting them, and be an excellent distraction while your sentry's mowing them down.

In your alcove case, I would try to place it against the wall towards the enemy, as if they walk by,

1. their field of view will either
   • not allow them to see it, so they will have to spend a split second aiming at it, all the while being laid into by it, or
   • if they are looking at it, then your team, further down the hall giving you support, will be able to fire at the enemy without challenge
2. the force from the bullets will drive the enemy further into the sentry's field of view, rather than in the "safe" retreating direction, towards the rest of their team

Here it is in ASCII because I'm too lazy to make it in paint.exe

Enemy ->           <- Your team
       ---       ---
         |       |  
         |X      |  
         |       |  
         ---------  

As the occasional soldier, whenever the second point is violated and there is insufficient support being given to the sentry, it's as good as dead because I can peek around the corner and peck it to death (or blow it all up at once as demo).

I assume that there is a corridor of sorts leading to the alcove and not a wide open space (otherwise placing a sentry in the alcove would make little sense to begin with).

What I usually do is place the sentry in the following location:

---      ---
  |   *  |  
  |     S|  
  |      |  
  --------  
  sentry at
  S, looking
  at *

Assuming both walls are not too far apart, this would allow the sentry to cover a slice of the hallway leading to its location, which is perfect since it would force your foes to a specific portion of the hallway if they hope to get the jump on the sentry. This, in turn, would allow your teammates or yourself to dispose of them with greater ease.

If the walls are far apart, however, I would go for the following placement:

---       ---
  |   *   |  
  |   S   |  
  |       |  
  ---------  
  sentry at
  S, looking
  at *

This would probably be the safest position to avoid grenades ricocheting off of the walls, and should make it harder to nail the sentry with stickies. I prefer to have the sentry looking forward so that it covers the entire access gap.

Wow, I forgot I never answered this question.

There are several criteria to take into account:

1. Soldier & Demoman: Build away from walls opposite an archway or from where your sentry can be hit by splash damage. Demoman can be a particular problem if they can arc stickies from below or above you. Don't forget you can destroy sticky bombs with your pistol or shotgun. Grenades also bounce around corners, so build on the near side of the corner rather than the far side.
2. Sniper: Don't build in front of an area with really long sight lines. While the Wrangler can mitigate this problem, controlling your sentry makes it much easier for the other team to sneak behind you.
3. Spy: Build somewhere where you can have your back to a wall. Listen for decloak sounds or your gun spotting something and turning towards you.
4. Uber: Well, there's really nowhere good to build a sentry to counter an Uber. Ubercharge was intended to be a hard counter to sentry guns when used effectively. Instead, it helps to have assistance from your teammates, particularly Pyros for their airblast ability. Also repair, repair, repair.

So, taking all this into account, your best bet is to build somewhere near two corners where the corners are a sufficient distance apart, but not to the point where you sentry can't see the other corner. A good example of this is Badwater Point D, which has two such areas like this for the RED team to put sentry guns. The one away from the tracks is more vulnerable to Demomen and Soldiers, the one closer to the tracks is more vulnerable to Spies and Ubers.

Now, there are some places where this just isn't possible, such as Granary Point B. In this case, your best bet is likely upstairs near the middle across from the point. In this position, your biggest problem will be enemies approaching from the right. A trick here is to place your dispenser behind the gun, and have the Engineer stand to its left (which is a dead end). This gives you some protection from Soldiers, Demomen,meaning that your main problem with be Spies and Ubers.

best place would be at a choke point like this:

----     -----
  |*      |    sentry at * and looking at s 
  | s     |  
  |       |  
  --------- 

Or if you are feeling sporty you could just place in a room/control point and keep a look out for the turret range (the gigantic red circle when holding the turret) and make sure the turret can cover all non-friendly entrences to the room/control point.