$n-$circular arrangement problem

As I understand it there are $n$ people, and among them $3$ given "special persons" $A$, $B$, $C$ that should not sit next to each other. (In your computation you only excluded the cases where all three sit together in adjacent seats.)

Take three chairs away from the table, and let $A$, $B$, $C$ hold them. Then seat the $n-3$ ordinary people arbitrarily. Considering the rotation symmetry this can be done in $(n-4)!$ ways. These $n-3$ people around the table leave between each other $n-3$ spaces. Each of these spaces can accept $\leq1$ special persons. $A$, $B$, $C$ select one of these spaces in turn. This can be done in $(n-3)(n-4)(n-5)$ ways.

The total number $N$ of allowed seatings therefore is $$N=(n-3)!\>(n-4)(n-5)\ .$$

Since 3 people must not sit together the no.of of ways to choose 3 people from $n$ is here $\binom{n}{3}$ and since we can arrange them in 2! ways (in a circle) it will be $\binom{n}{3} 2!$ . And the remaining no.of of people are $n-3$ and since there are 3 spaces between 3 people (on a circle) so no.of solutions of $x_1 + x_2 + x_3 = n-3 , x_1 , x_2 , x_3 \geq 1 \Rightarrow $ solutions of $ y_1 + y_2 + y_3 = n-6$ viz. $\binom{n-4}{2}$ and the number of ways to arrange $n-3$ objects in a row is $(n-3)!$. So finally we get $ \binom{n}{3} 2! (n-3)! \binom{n-4}{2}$.

I hope this answer is fine. Any tiny errors, correct me!!

Comment of @Christian Blatter might be helpful to you in counting the missing cases. However, here's an alternate method.

Let the three people be $P_1,\ P_2,\ P_3$ and the number of people between $P_1P_2,\ P_2P_3,\ P_3P_1$ be $x_1,\ x_2,\ x_3$ respectively. Now, we've to find number of positive integral solutions of the equation $$x_1+x_2+x_3=n-3$$ Also, the $n-3$ people can arrange themselves in $(n-3)! $ ways and the three people in $2! $ ways.

The total no. of ways are $\displaystyle{{n-4}\choose 2}×(n-3)!×2! =(n-3)!(n-4) (n-5) $.