The hot hand and coin flips after a sequence of heads

ESPN recently posted a story demonstrating that the "hot hand" concept is, in fact, real. Part of the justification is this example based on coin flips from a paper by Adam Sanjurjo and Joshua B. Miller:

And now [Joshua] Miller brings it back to coin flips and the subtle selection bias of hit streaks. The broken-clock example, Miller says, is an extreme illustration of what's happening in a long line of coin flips. Let's say that in analyzing the results of 100 coin flips you see a string of three heads (HHH_) and "_" is, say, Flip 42 in the sequence -- which was selected because it was preceded by HHH. What is the _ in the sequence? Of course, you don't know which kind of sequence you are in. it could be HHHT or it could be HHHH. You think it's a 50-50 chance that Flip 42 is H (or T).

But here's where it changes, and it is sneaky. In sequence HHHH, you could have selected Flip 43 because it continues a run of three heads, and you haven't excluded Flip 44 or 45, like you would have in sequence HHHT (it ends the run of heads). The excluded flips in the HHHT world mean it is more likely that Flip 42 will turn up T rather than H, given the condition that the flip was selected because it was preceded by three heads in a row. The choice has been restricted. To tie it back to the big ol' mansion, the broken clock excludes more times that are different from the one you are seeing, and the kind of sequence with HHHT excludes more flips that are different from the one you selected.

I am struggling to follow this explanation and the significant of the restricted choice. What does excluding flips 44 or 45 mean? Why is flip 42 more likely to be T rather than H?

To answer your question about the significance of restricted choice, and unpack that explanation, let's consider a simpler version of the example that Adam Sanjurjo and I shared with ESPN (we are following spaceisdarkgreen's lead here, but sticking with the approach in the ESPN article).

Imagine flipping a coin 3 times, and then selecting at random one of the flips that is immediately preceded by an $H$. You will select either flip 2 or flip 3, with equal chance. Let's say you selected flip 2 (analogous to flip 42). Well you know flip 1 is an $H$, because "preceded by $H$" was the selection rule. So you know you are in a sequence in which the outcomes of flip 1 and 2 are either $HH$, or $HT$. Why do we know $HT$ is more likely? Well, before flipping, the odds in favor of $HT$ relative to $HH$ were $1:1$, but the likelihood of selecting flip 2 (the flip we see) in world $HT$ is twice as likely when compared to world $HH$, because in world $HT$ you have to select flip 2 (i.e. $\mathbb{P}(flip\ 2|HT)=1$), whereas in world $HH$, you could selected flip 3 (i.e. $\mathbb{P}(flip\ 2|HH)=1/2$). So ex post, the odds in favor of $HT$ (relative to $HH$) double to $2:1$. With $2$ chances of $HT$ out of $3$ total chances, the probability of $HT$ becomes $2/3$, i.e. the probability flip 2 is a tails is $2/3$. This is Bayes rule, in odds form, which is simpler because it involves only multiplication. Dropping the verbiage, we have:

\begin{align} \frac{\mathbb{P}(HT|flip\ 2)}{\mathbb{P}(HH|flip\ 2)}& = \frac{\mathbb{P}(flip\ 2|HT)}{\mathbb{P}(flip\ 2|HH)}\times \frac{\mathbb{P}(HT)}{\mathbb{P}(HH)} \\ & = \frac{1}{1/2}\times\frac{\mathbb{P}(HT)}{\mathbb{P}(HH)} \\ & = 2\times\frac{1/2}{1/2} \\ & = 2 \end{align}

This gives the intuition behind the bias (for flips that are not the final flip). You are more likely to select the flip you selected, flip 2, in the world in which it is a $T$, because in that world you have to, whereas in the world in which it is an $H$, you are less restricted in your choice as there are more other flips to choose from -- you could have selected the next flip, flip 3.

Note that if flip $3$ were selected, than for the world in which flip 2 and 3 are $HT$, the likelihood of you selecting flip 3 is the same as in the world in which flip 2 and 3 are $HH$, because $HT$ doesn't exclude flip 4 as a possibility, becase there is no flip 4 (we can ignore the first flip here). That means the odds don't change.

So considering all three flips, we have found that $\mathbb{P}(HT|flip\ 2) =2/3$ and $\mathbb{P}(HT|flip\ 3) =1/2$. Because each flip is equally likely to be chosen we have

$$\mathbb{P}(T|\text{flip preceded by $H$})= 1/2\times 2/3 + 1/2\times 1/2 = 7/12$$

Note 1: The ESPN example is from this general interest write-up of ours---while more complicated with longer streak lengths, it gives intuition about strength of bias as streaks get longer. The simpler version presented here can be found in a paper by Adam Sanjurjo and myself, which connects restricted choice to the hot hand and other conditional probability problems. We borrowed one popular way of communicating the intuition for why, in the classic Monty Hall problem, the contestant should update his/her beliefs when a door is opened (see this StackExchange for the details).

Note 2: The response of spaceisdarkgreen is a good one, along with the extra details in the comments, but it doesn't answer Barry's specific question. Spaceisdarkgreen's focus is on the sampling distribution of the proportion and the calculation certainly demonstrates that the measure is biased, but it doesn't give an intuition for the direction of the bias.

Forget $42-45...$ the same effect should be present in, say, three coin flips where the "streak" is one head in a row. Say you are told "I've randomly selected a heads flip that is not the final (3rd) flip." What is the probability that the following flip is heads? There's an issue here... It could easily be the case that there is no heads meeting that criterion. In fact it only happens when one of the first two flips is heads. So we are equally likely to have $HHH$, $HHT$, $HTH$, $HTT$, $THH$, $THT.$ The probabilities for having tails be the next flip after you randomly select a heads from the first two are $0,1/2,1,1,0,1$ respectively, so the probability of tails is $3.5/6 > .5$