Combination and permutation of indistinguishable objects
To explain where your formula comes from and why it works, consider breaking it up into steps:
Step 1: choose where the reds go. There are $\binom{10}{2}$ ways of arranging the reds and not reds (ignoring the fact that the not reds are of multiple colors for the moment).
Step 2: of the spaces labeled for use by not-reds, choose which of those spaces will be occupied by blues: There are $\binom{10-2}{3}$ number of ways to do this.
Step 3: of the spaces labeled for use by not-reds and not-blues, choose which are occupied by greens: There are $\binom{10-2-3}{5}$ number of ways.
Thus, there are $\binom{10}{2}\cdot\binom{8}{3}\cdot\binom{5}{5} = \frac{10!~~~8!~~~5!}{2!8!3!5!5!0!} = \frac{10!}{2!3!5!}$ number of ways to accomplish this.
(remember that $0!=1$ by definition)