Why isn't there only one way of painting these horses?

To understand the question you need to know something about the general behaviour of horses.

It is a well established fact that horses like to "horse around". What this tells us is that the horses are all positioned in a circular fashion, we need to find the ways to paint the horses, so that rotation of an arrangement counts as the same arrangement.

Now, if they where in a line the answer would be $\binom{11}{5,3,3}$. But each of these "linear" arrangements gives way to $11$ circular arrangements. Since closing the line and rotating it gives the $11$ arrangements.( to see this it is important to note $5$ and $3$ are both relatively prime to $11$).

Hence the answer is $\frac{1}{11}\binom{11}{5,3,3}$ as desired.