An elementary way to show that the determinant is non zero

Solution 1:

This might be equivalent to what you did, but I'll go ahead and write this as an answer anyway.

Let $X = \begin{bmatrix}0 & 0 & -1 \\ 1 & 0 & -2 \\ 0 & 1 & 0\end{bmatrix}$. Then, $X^2 = \begin{bmatrix}0 & -1 & 0 \\ 0 & -2 & -1 \\ 1 & 0 & -2\end{bmatrix}$, and so, $$Y := \begin{bmatrix}a & -c & -b \\ b & a - 2c & -c -2b \\ c & b & a -2c\end{bmatrix} = aI+bX+cX^2.$$

The eigenvalues of $X$ are the three roots $\lambda_1,\lambda_2,\lambda_3$ of $\lambda^3+2\lambda+1 = 0$.

Since $Y = aI+bX+cX^2$, the eigenvalues of $Y$ are $a+b\lambda_k+c\lambda_k^2$ for $k = 1,2,3$.

Now, suppose $\det(Y) = 0$ for some integers $a,b,c$ with $abc \neq 0$. Then, $0$ must be an eigenvalue of $Y$, and so, $a+b\lambda_k+c\lambda_k^2 = 0$ for some $k \in \{1,2,3\}$. But by using the quadratic formula, we have $\lambda_k = \dfrac{-b \pm \sqrt{b^2-4ca}}{2c}$.

So now you just need to prove that none of the three roots of $\lambda^3+2\lambda+1 = 0$ can be expressed in the form $\dfrac{-b \pm \sqrt{b^2-4ca}}{2c}$ for non-zero integers $a,b,c$. This just requires showing that $\lambda^3+2\lambda+1$ is irreducible in $\mathbb{Z}[\lambda]$, which only requires the rational root theorem and checking that $\pm 1$ are not roots.

Solution 2:

Say we have a counterexample. If $3$ divides each of $a,b,c$, then we can divide each by $3$ without consequence, and obtain another counterexample. We may do this until $3$ fails to divide one of $a,b,c$.

However, we may calculate that $3$ fails to divide the determinant in any other case. To do this we would ordinarily have $26$ cases. However, since our polynomial is homogeneous in $a,b,c$, we only need to check triples up to scaling, which halves the number of cases. When two of $\{a,b,c\}$ are $0\bmod 3$ this check is easy, as only a single nonzero term persists in the determinant expansion. This reduces it to only $10$ cases, each of which can be checked by hand (it may even be possible to reduce further without much computation).

Remark. This strategy works for any prime $p$ for which $t^3+2t+1$ is irreducible in $\mathbb F_p[t]$. This isn't true for $p=2$, so $p=3$ yields the fewest number of cases.