Find $xy+yz+zx$ given systems of three homogenous quadratic equations for $x, y, z$

We can obtain $yz+zx+xy=2$ simply by finding the values of $x$, $y$, and $z$. We are given $$y^2+yz+z^2=1,\qquad(1)$$$$z^2+zx+x^2=4,\qquad(2)$$$$x^2+xy+y^2=3,\,\qquad(3)$$with $x,y,z>0$. Subtracting eqn $3$ from eqn $2$, and noticing the factor $z-y$ on the LHS, gives $$(z-y)(x+y+z)=1.$$Similarly, from eqn $2$ minus eqn $1$, we get$$(x-y)(x+y+z)=3.$$The ratio of the latter two equations yields $x-y=3(z-y)$, or$$x=3z-2y.$$ Now substituting for $x$ from the above equation into eqn $3$ yields$$y^2-3yz+3z^2=1.$$By subtracting this from eqn $1$, we find $2yz-z^2=0$, or$$z=2y$$(since $z\neq0$). Substitution for $z$ into eqn $1$ now gives $7y^2=1$. It follows that $y=1/\surd7$, because $y>0$, and then $z=2/\surd7$ and $x=4/\surd7$, giving$$yz+zx+xy=2.$$

To my surprise, this problem can be solved using geometry.

Identify the Euclidean plane with complex plane $\mathbb{C}$. and let $\omega = e^{\frac{2\pi}{3}i}$ be the cubic root of unity.

Consider triangle $\triangle ABC$ with vertices at $A = x$, $B = y\omega$ and $C = z\omega^2$.
The sides of the triangle equal to

$$\begin{align} a^2 = BC^2 &= | y\omega - z\omega^2|^2 = y^2 + yz+ z^2 = 1\\ b^2 = AC^2 &= | x - z\omega^2|^2 = x^2 + xz + z^2 = 4\\ c^2 = AB^2 &= | x - y\omega|^2 = x^2+xy+y^2 = 3 \end{align} $$ Since $a^2 + c^2 = b^2$, this is a right angled triangle with area $\mathcal{A} = \frac12 ac =\frac{\sqrt{3}}{2}$.

An alternate way to compute the area is cut the triangle into 3 pieces along the line $OA$, $OB$ and $OC$. This gives us

$$\mathcal{A}= \frac12 (xy + yz + xz)\sin\frac{2\pi}{3} = \frac{\sqrt{3}}{4} (xy+yz+xz)$$ Combine these two results, we find: $$xy+yz+xz = 2$$

Update

For a pure algebraic answer, one can substitute above expression of $a^2,b^2,c^2$ into Heron's formula for area of triangle, $$\mathcal{A} = \frac14\sqrt{(a^2+b^2+c^2)^2-2(a^4+b^4+c^4)}$$ simplify and obtain following algebraic identity

$$\begin{array}{rl} 3(xy+yz+zx)^2 =& \phantom{+0}((x^2+xz+y^2) + (y^2+yz+z^2) + (z^2+xz+x^2))^2\\ &-2((x^2+xz+y^2)^2 + (y^2+yz+z^2)^2 + (z^2+xz+x^2)^2) \end{array} $$ Using given values of $a, b, c$, we find $$3(xy+yz+xz)^2 = (3+4+1)^2 - 2(3^2+4^2+1^2) = 64 -2(26) = 12 = 3(2)^2$$ This leads to the same conclusion $xy+yz+xz = 2$ as before.

I would like to mention an other method, more algorithmic but also much more general to this problem.

First compute a Gröbner basis (say, using the lexicographic order) of the three polynomials $$x^2 + xy + y^2 - 3 \\ y^2 + yz + z^2 - 1 \\ x^2 + xz + z^2 - 4$$ One obtains $\left\{7 z^3-4 z,2 y z-z^2,2 y^2+3 z^2-2,x+2 y-3 z\right\}$.

We want to evaluate $xy+yz+zx$, which is presumably a constant, under the given conditions. This means that $xy+yz+zx-a$ belongs to the ideal generated by the three above polynomials, for some constant $a$ to be determined. Let's evaluate the remainder of the Euclidean division of $xy+yz+zx-a$ by the Gröbner basis given above: one finds $$-2-a+7z^2 \, . $$ This should be zero, if our relation is to be a consequence of the original relations. So we deduce $a=7z^2-2$.

What is the value of $z^2$? Well, the first term of the Gröbner basis tells us that $z(7z^2-4)=0$, so $z^2 = 0 , 4/7$, and therefore $a= \pm 2$.

To fix the sign, we need the last piece of information, namely that $x,y,z \geq 0$. Clearly, if $z=0$ then the last term of the basis tells us that $x+2y=0$, which is forbidden (since $x=y=z=0$ is not a solution). Therefore, $z^2 = 4/7$ and $a=2$.