$A\in M_3(\mathbb R)$ orthogonal matrix with $\det (A)=1$. Prove that $(\mathrm{tr} A)^2- \mathrm{tr}(A^2) = 2 \mathrm{tr} (A)$

$\begin{align*}\mathrm{Tr}(A)^2-\mathrm{Tr}(A^2)&= (\lambda_1+\lambda_2+\lambda_3)^2-(\lambda_1^2+\lambda_2^2+\lambda_3^2)\\&=2(\lambda_1\lambda_2+\lambda_2\lambda_3+\lambda_3\lambda_1)\\&=2\lambda_1\lambda_2\lambda_3\left(\frac{1}{\lambda_1}+\frac{1}{\lambda_2}+\frac{1}{\lambda_3}\right)\\&=2\det(A)\mathrm{Tr}(A^{-1})\\&=2\mathrm{Tr}(A). \end{align*}$

(Edit: In the last step, we use the facts that $A^{-1}=A^T$ when $A$ is orthogonal and $\mathrm{Tr}(A^T)=\mathrm{Tr}(A)$ for any matrix $A$.)

In $\mathbb Z[X,X^{-1}]$ we have $$(1+X+X^{-1})^2-(1+X^2+X^{-2})=2+2X+2X^{-1}.$$ In other words, your equality holds over any commutative ring, for any matrix similar to a diagonal matrix with diagonal entries $1,a,a^{-1}$, where $a$ is a unit.

An old-fashioned argument

The matrix $A$ is the matrix of a rotation through some angle $\theta$. It is a standard result that the trace of such a rotation is $1+2\cos\theta$. The matrix $A^2$ is a rotation through $2\theta$ about the same axis.

So we want to show that $$(1+2\cos\theta)^2-(1+2\cos 2\theta)=2(1+2\cos\theta).$$

With a little manipulation this reduces to the familiar $\cos 2\theta=2\cos^2\theta -1$.

A rotation is diagonalisable over the complex numbers, but generally not over the reals. However, there always is an orthonormal basis $\mathcal{B}$ and a real number $\theta$ such that $$\mathrm{Mat}_{\mathcal{B}}(A)=\left(\begin{array}{ccc} \cos(\theta) & -\sin(\theta) & 0 \\ \sin(\theta) & \cos(\theta) & 0 \\ 0&0&1\end{array}\right)$$ and you'll have $$\mathrm{Mat}_{\mathcal{B}}(A^2)=\left(\begin{array}{ccc} \cos(2\theta) & -\sin(2\theta) & 0 \\ \sin(2\theta) & \cos(2\theta) & 0 \\ 0&0&1\end{array}\right)$$ Thus $\mathrm{Tr}(A)=1+2\cos(\theta)$ and $\mathrm{Tr}(A^2)=1+2\cos(2\theta)$. The rest follows from trigonometry.

Hint: $$\det(I-A)=\det(A^tA-A)=\det(A^t-I)\det(A)=\det(A-I)\cdot1=(-1)^3\det(I-A),$$ so $\det(I-A)=0$. IOW one of the eigenvalues is equal to $1$. Does that help?