Solving a system of non-linear equations

Given $x^2-yz = 1, \quad y^2-xz = 2, \quad z^2-xy = 3$, we can sum all of these to get $$(x-y)^2+(y-z)^2+(z-x)^2 = 12 \tag{1}$$

OTOH, subtracting gives $(y^2-x^2)+z(y-x)=1 \implies (x+y+z)(y-x) = 1$ and similarly $(x+y+z)(z-y) = 1$, so we must have $y-x = z - y = a$, say. Using this in $(1)$, $$a^2+a^2+4a^2=12 \implies a = \pm \sqrt2$$ So we have $y = x \pm \sqrt2, \quad z = x \pm 2\sqrt2$. Using these in say the first equation, you should be able to solve for $x$ and then $y, z$.

Let's rewrite the system as $$\mathbf r \times \mathbf P\mathbf{r} = \begin{vmatrix}\mathbf{i}&\mathbf{j}&\mathbf{k}\\x&y&z\\z&x&y\end{vmatrix} = 2\mathbf{i} + 3\mathbf{j} + 1\mathbf{k} \equiv \mathbf f$$ where $\mathbf{r} = (x,y,z)^\top$ and $$ \mathbf P = \begin{pmatrix} 0 & 0 & 1\\ 1 & 0 & 0\\ 0 & 1 & 0 \end{pmatrix} $$ So we want to solve $$ \mathbf{r} \times \mathbf{Pr} = \mathbf{f}. $$ Let's dot multiply both sides with $\mathbf{r}$ $$ 0 = \mathbf{r}^\top \mathbf{f} $$ also when multiplying by $\mathbf{Pr}$: $$ 0 = \mathbf{r}^\top \mathbf P^\top \mathbf{f} $$ That's a pair of linear equations in $x,y,z$. Since the equations are homogeneous, the solution will be of the form $\mathbf r = \alpha \mathbf{h}, \alpha \in \mathbb{R}$. $$ 2x + 3y + z = 0\\ 2z + 3x + y = 0 $$ Row reducing the system we get $$ \mathbf h = \begin{pmatrix} -5\\1\\7 \end{pmatrix}. $$ Since $\mathbf h \times \mathbf {Ph} = 18 \mathbf f$ we deduce that $$ \alpha^2 = \frac{1}{18}\qquad\alpha = \pm\frac{1}{3\sqrt{2}}. $$ Finally $$ x = \mp \frac{5}{3\sqrt{2}}\\ y = \pm \frac{1}{3\sqrt{2}}\\ z = \pm \frac{7}{3\sqrt{2}} $$

The resultant of $x^2-yz-1$ and $y^2-xz-2$ with respect to $z$ is $x^3-y^3-x+2y$. The resultant of $x^2 - yz - 1$ and $z^2 - xy - 3$ with respect to $z$ is $x^4-x y^3-2 x^2-3 y^2+1$. The resultant of $x^3-y^3-x+2y$ and $x^4-x y^3-2 x^2-3 y^2+1$ with respect to $x$ is $-y^4(18 y^2-1)$. So either $y = 0$ or $y = \pm 1/\sqrt{18}$.

With $y=0$ we get $x^2 - 1 = 0$, $-xz-2 = 0$ and $z^2 - 3 = 0$, which clearly will not work.

With $y = \pm 1/\sqrt{18} = \sqrt{2}/6$ we do get solutions: $x = \mp 5 \sqrt{2}/6$, $z = \pm 7 \sqrt{2}/6$.

Note: This is a slightly clumsy but systematic approach. On the plus side, this allow you solving similar equations of the form $$\begin{cases} x^2 - Ayz &= D\\ y^2 - Bxz &= E\\ z^2 - Cxy &= F \end{cases}$$ without knowing how to complete the squares. On the minus side, you need to factor a quartic polynomial in the middle of the process.

Notice the LHS of given set of equations are all homogenous with degree $2$, $$\begin{cases} x^2 - yz &= 1\\ y^2 - xz &= 2\\ z^2 - xy &= 3 \end{cases}\tag{*1}$$ we can simplify it by looking at the ratios first. i.e. let $u = \frac{y}{x}$ and $v = \frac{z}{x}$, we have

$$ \begin{cases} \frac{1-uv}{1} &= \frac{u^2 - v}{2}\\ \frac{1-uv}{1} &= \frac{v^2 - u}{3} \end{cases} \iff \begin{cases} 2(1-uv) &= u^2 - v\\ 3(1-uv) &= v^2 - u \end{cases} \implies \begin{cases} v &= \frac{2-u^2}{2u-1} &(*2a)\\ u &= \frac{3-v^2}{3v-1} &(*2b) \end{cases} $$ Substitute $(*2a)$ into $(*2b)$, we get

$$u = \frac{3 - \left(\frac{2-u^2}{2u-1}\right)^2}{3\left(\frac{2-u^2}{2u-1}\right) - 1} \iff \frac{5u^4+u^3-5u-1}{6u^3+u^2-16u+7} = \frac{(u-1)(5u+1)(u^2+u+1)}{(2u-1)(3u^2+2u-7)} = 0\\ $$ Since $u^2 + u + 1 = (u+\frac12)^2 + \frac34 > 0$ for all real $u$, there are only two choices for $u$:

• Case 1 : $u = 1 \implies v = \frac{2 - 1^2}{2-1} = 1 \implies 1 - uv = 0$.

  However, this contradicts with the requirement $x^2 - yz = x^2(1-uv) = 1$, this doesn't lead to any solution for $(*1)$.

• Case 2 : $u = -\frac15 \implies v = \frac{2 - \left(-\frac15\right)^2}{2\left(-\frac15\right)-1} = -\frac75 \implies 1 - uv = \frac{18}{25}$.

  We have $x = \frac{1}{\sqrt{1-uv}} = \pm\frac{5}{3\sqrt{2}}$ now. This leads to two real solutions for $(*1)$. $$(x,y,z) = (x,xu,xv) = \left(\pm\frac{5}{3\sqrt{2}}, \mp\frac{1}{3\sqrt{2}}, \mp\frac{7}{3\sqrt{2}} \right) $$