How does a sentry prioritize targets?

             H
      {o}

      D      S

The S in this case is the sentry. The D is a demoman, running towards sentry with Grenade Launcher. The H is a heavy, firing at the sentry. The o is a scout, and he is concurrently capping a point ({}signifies a cap point)

Will the sentry shoot the scout first? Demoman first? How does it pick its targets? Is it random?

Whoever is closest to the Sentry Gun within its range will be targeted by it. If the target moves away such that another target becomes the closest, the Sentry Gun will switch to that next closest target. This is why during an ÜberCharge, it's recommended that a Medic stand between his patient and the sentry to tank its gunfire, so that his patient can damage it.

Highly-coordinated team movements could theoretically cause a sentry to turn back and forth without allowing it to open much fire and cause too much damage, but given a typical game's fast pace, I haven't observed that happen yet.

Note that it doesn't matter what a target is doing; only their position relative to the Sentry Gun matters. Whether the Scout is on the control point or not, if he's the closest to the Sentry Gun then it will fire at him. Whether the Demoman or Heavy is firing at it, whoever is closest will be targeted.

Also, if there's a Spy using Your Eternal Reward, then as long as he is disguised he will never be targeted by a Sentry Gun, even if he backstabs somebody while already disguised and within its range.