Number of triangles in a regular polygon

Put an admissible triangle on the regular polygon, and consider a string of the $n$ vertices, enumerated in a circular fashion; we denote the vertices of the triangle by $X$, and the $n-3$ other ones by $I$.

We know that $\ldots XI \ldots XI \ldots X \ldots$ must be part of the string, since the triangle's vertices are non-adjacent. We see that the $X$'s separate the string into $4$ distinguishable regions. Now we put the $n-5$ remaining vertices, all indistinguishable from one another, into the regions of the string. This can be done in $n-5+3 \choose 3$ different ways.

However, the outer regions of the string cannot both be empty, so we have to exclude some possibilities. This situation is similar to the earlier one, only that there are now just $2$ regions available (namely, the middle ones). Hence we have $n-5+1 \choose 1$ cases to subtract from the above.

In consequence, the answer is ${n-5+3 \choose 3} - {n-5+1 \choose 1}$. It works for all integers $n$, too!

We first count the triangles that have a specific vertex $P$.

For the other vertices of our triangle, we cannot use the two neighbours of $P$. That leaves $n-3$ vertices. We have to choose $2$ of them, but they must not be neighbours.

There are $\binom{n-3}{2}$ ways to choose $2$ vertices. But $n-4$ of these choices give us a pair of neighbours. So the number of "good" choices is $\binom{n-3}{2}-(n-4)$. This simplifies to $\frac{(n-4)(n-5)}{2}$.

Now multiply by $n$ to take account of all possible choices for $P$. However, this overcounts. Every triangle $ABC$ is counted $3$ times, once for having $A$ as a vertex, once for $B$, and once for $C$. Thus the number of triangles is $$\frac{n(n-4)(n-5)}{6}.$$