Using Lagrange's diagonalization on degenerate linear forms
Let $A=\begin{pmatrix}1 & 2 & 3\\ 2 & 3 & 4\\ 3 & 4 & 5 \end{pmatrix}$ be a real matrix. Find an invertible matrix $P\in M_{3}(\mathbb{R})$ such that $P^TAP$ is diagonal whose elements on the diagonal are all in the set $\{-1, 0, 1\}$
I've been trying to solve this with Lagrange's squares method, but I'm stuck at the end where I seem to get "less" squares then required. I assume that this is because $A$ is degenerate, but I'm unable to complete my form to a basis that would satisfy the requirements as a result, and would love a general method for this.
These are my calculations so far:
Let $\begin{pmatrix}x_{1}\\ x_{2}\\ x_{3} \end{pmatrix}\in\mathbb{R}^{3}$. Then
$$\begin{aligned}\begin{pmatrix}x_{1}\\ x_{2}\\ x_{3} \end{pmatrix}^{T}\begin{pmatrix}1 & 2 & 3\\ 2 & 3 & 4\\ 3 & 4 & 5 \end{pmatrix}\begin{pmatrix}x_{1}\\ x_{2}\\ x_{3} \end{pmatrix}= & x_{1}^{2}+4x_{1}x_{2}+6x_{1}x_{3}+3x_{2}^{2}+5x_{3}^{2}+8x_{2}x_{3}\\ = & x_{1}^{2}+2x_{1}\left(2x_{2}+3x_{3}\right)+\left(2x_{2}+3x_{3}\right)^{2}\\ & -\left(2x_{2}+3x_{3}\right)^{2}+3x_{2}^{2}+5x_{3}^{2}+8x_{2}x_{3}\\ = & \left(x_{1}+\left(2x_{2}+3x_{3}\right)\right)^{2}-4x_{2}^{2}-12x_{2}x_{3}-9x_{3}^{2}+3x_{2}^{2}+5x_{3}^{2}+8x_{2}x_{3}\\ = & \left(x_{1}+2x_{2}+3x_{3}\right)^{2}-x_{2}^{2}-4x_{2}x_{3}-4x_{3}^{2}\\ = & \left(x_{1}+2x_{2}+3x_{3}\right)^{2}-\left(x_{2}+2x_{3}\right)^{2} \end{aligned} $$
and you can see I have only two "elements" for the basis, and I'm stuck..
Denote the quadratic form by
$$ q \left( \begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} \right) = (x_1, x_2, x_3) \cdot A \cdot \begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} = (x_1 + 2x_2 + 3x_3)^2 - (x_2 + 2x_3)^2. $$
We see that if we make the change of variables
$$ u_1 = x_2 + 2x_2 + 3x_3, \,\, u_2 = x_2 + 2x_3, u_3 = r_{31} x_1 + r_{32} x_2 + r_{33} x_3 $$
the quadratic form will become diagonal. You can choose the scalars $r_{31}, r_{32}, r_{33}$ in any way you like as long as you get a legitimite change of variables - that is, the corresponding map is invertible.
More formally, consider the matrix $R$ given by
$$ R = \begin{pmatrix} 1 & 2 & 3 \\ 0 & 1 & 2 \\ r_{31} & r_{32} & r_{33} \end{pmatrix}, \,\, R \begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} = \begin{pmatrix} u_1 \\ u_2 \\ u_3 \end{pmatrix}. $$
If we choose $r_{31}, r_{32}, r_{33}$ so that $R$ will be invertible, then
$$ q \left( R^{-1} \begin{pmatrix} u_1 \\ u_2 \\ u_3 \end{pmatrix} \right) = (u_1, u_2, u_3) \left( R^{-1} \right)^T A R^{-1} \begin{pmatrix} u_1 \\ u_2 \\ u_3 \end{pmatrix} = u_1^2 - u_2^2 $$
and so the $P$ you are looking for can be taken to be $R^{-1}$. The simplest choice (that will also make computing $R^{-1}$ easier) is to take $r_{31} = r_{32} = 0$ and $r_{33} = 1$.