Number of triangles sharing all vertices but no sides with a given octagon

If two of the vertices are $A$ and $C$, what are the possible third vertex? Look at the whole list $A,...,H$

Suppose the vertices are labelled $1,2,\dots,8$. Count the number of triangles for which one of the vertices is $1$. Then the second vertex, going around "clockwise" (let's say the octagon was represented that way) is among $3,4$ or $5$ (if we put one at $6$, the third vertex would be $7$ or $8$, which would give the triangle a side in common with the octagon). For each case you can count the number of options : three options for $3$, two for $4$, one for $5$, for a total of $6$. This gives us $6$ triangles that have a vertex at $1$.

The group $\mathbb Z / 8 \mathbb Z$ acts on the triangles by mapping the triangle with vertices $(a,b,c)$ to the triangle with vertices $(a+k,b+k,c+k)$ (where $k \in \mathbb Z / 8 \mathbb Z$ and you can consider $a,b,c \in \mathbb Z / 8 \mathbb Z$). It is not hard to see that each triangle has an orbit of size $8$ under this action.

So if we count the triangles by considering those who have a vertex at $1$ and then rotate them via the group action, we will triple count because each triangle has three vertices. Therefore the answer is $(6 \times 8)/ 3 = 16$.

Hope that helps,