Binomial probability doesn't give the correct number?

First off, note that $$\frac{1}{6^5} = \frac{1}{7776} \approx 0.013\%$$ so your first computation is incorrect. It is true that $\approx 0.08\%$ of the time that all five dice rolls will match one another, but this isn't given by $1/6^5$. Rather, it's given by $6/6^5 = = 1/6^4$.

The reason for this has been given in the comments, but in short, it's because there are $6$ ways for all five dice rolls to match. That is, all five could come up 1, all five could come up 2, all five could come up 3, and so on for each of the six faces. Therefore, out of the $6^5$ total equally likely results for the five dice rolls, $6$ result in all five matching, giving the correct probability of $6/6^5 = 1/6^4$.

Now I'll address the major issue in your formulation of the binomial probability: Your definition of $r$ is deficient.

• $r$ is the number of dice you need to match

to match... what? To be correct, it should be

• $r$ is the number of dice you need to match a fixed value

For instance, you could define $r$ as the number of dice that come up $6$. Then the probability that all $5$ rolls come up $6$ would be given as the binomial probability $$P=\frac{5!}{(5-5)!5!}\times(\frac{1}{6})^{5}\times(1-\frac{1}{6})^{5-5}\approx0.012\%$$ with no error.