Help solving $ax^2+by^2+cz^2+dxy+exz+fzy=0$ where $(x_0,y_0,z_0)$ is a known integral solution
I think I ought to start by pointing out that we find all primitive solutions to $x^2 + y^2 - z^2 = 0$ (Pythagorean triples) using a parametrization in two variables, $x=r^2 - s^2, y = 2 r s, z = r^2 + s^2,$ with $\gcd(r,s) = 1$ and one odd, one even. This type of $2$ variable outcome is evident when, in the original question, $d=e=0,$ so that we have $a x^2 + d x y + b y^2 = - c z^2.$ We simply define rational variables $u = x/z$ and $v = y/z.$ The hypothesis is that there is at least one rational point $(u_0, v_0)$ on the conic section $a u^2 + d u v + b v^2 = -c.$ We find all rational points by stereographic projection, $u = u_0 + t u_1,$ $v = v_0 + t v_1.$ Plug into the defining equation for the conic section, take $t \neq 0$ gives another, nonzero value for $t,$ plug that into the projection, clear denominators, the result is a parametrization by two variables.
With $ax^2+by^2+cz^2+dxy+exz+fzy=0,$ we may use stereographic projection around a given integer solution. This results in a three parameter description of solutions. It gives a multiple of every primitive solution. It is sometimes a little hard to predict the GCD of the three variables, meaning it is hard to be sure we have found all solutions with given bounds on the entries.
There is an interesting alternative, method goes back to Fricke and Klein, gives a two variable parametrization, and can be adjusted to deal with GCD's. There is a complete answer to this, finding all primitive solutions, meaning $\gcd(x,y,z) = 1.$
We begin with finding all primitive solutions to $y^2 - z x = 0.$ If $g = \gcd(x,z) > 1,$ then $g^2 | y^2$ and $g | y,$ so $g | \gcd(x,y,z).$ However, $\gcd(x,y,z) = 1.$ So, $\gcd(x,z) = 1.$ Since $xz = y^2,$ either $x=u^2, z=v^2,$ or $x=-u^2,z=-v^2,$ in either case with $\gcd(u,v) = 1.$ That is, possibly by changing from $(x,y,z)$ to $(-x,-y,-z)$ so as to arrange $x \geq 0,$ all primitive solutions are $$ x = u^2, y = u v, z = v^2. $$
Next, your quadratic form is $X^T G X / 2,$ where $$ G = \left( \begin{array}{rrr} 2a & d & e \\ d & 2b & f \\ e & f &2c \end{array} \right) $$ and $$ X = \left( \begin{array}{r} x \\ y \\ z \end{array} \right) $$
The quadratic form $y^2 - z x$ is $X^T H X / 2,$ where $$ H = \left( \begin{array}{rrr} 0 & 0 & -1 \\ 0 & 2 & 0 \\ -1 & 0 & 0 \end{array} \right) $$ It is a theorem in Fricke and Klein (1897), pages 507-508 that, because your quadratic form has at least one integer solution, in which $(x,y,z)$ are not all zero, there exists a square matrix of integers $R$ and a nonzero integer $n$ such that $$ R^T G R = n H. $$ As $R$ has an inverse, we can take $S$ to have integral entries and minimize positive $k$ in $$ RS = SR = k I. $$
We already know that we can take all solutions of $y^2 - z x = 0$ as the column vector $$ U = \left( \begin{array}{r} u^2 \\ uv \\ v^2 \end{array} \right) $$ for relatively prime $(u,v).$ That is, $U^T H U = 0,$ and all solutions are a scalar multiple of $U.$
What happens if $X^T G X = 0,$ the "solutions" we want, with gcd one? Well, $R^T G R = n H,$ so $S^T R^T G R S = n S^T H S,$ so $$ G = \frac{n}{k^2} S^T H S, $$ and $X^T G X = 0$ says $$ X^T S^T H S X = 0. $$ We have already shown that there is some integer $w$ with $$ SX = w U. $$ This gives us $RSX = w RU$ and $kX = w RU.$ Now, as $\gcd(x,y,z) = 1,$ there is a row vector $A = (\alpha,\beta,\gamma)$ with $AX = 1.$ This tells us $k = w ARU.$ As $ARU$ is some integer, $w | k,$ and the earlier $kX = w RU$ becomes $$ X = \frac{1}{h} R U $$ for $h = \frac{k}{w} \in \mathbb Z.$ Furthermore, as $ RS = SR = k I, $ we know $k | \det R,$ so $h | k$ tells us $h | \det R.$ One may leave it this way: list the divisors of $\det R,$ including $\det R$ itself. For each primitive pair $(u,v),$ produce the column vector $RU,$ which will be a solution but perhaps not primitive. Divide out by the gcd of the entries of $RU.$ All integer primitive solutions are given by $$ X = RU/ g_1, $$ where $g_1$ is the gcd of the three entries of $RU.$ It is worth emphasizing that $g_1$ is a divisor of $\det R.$ Also, we get some explicit bounds, as $$ |X|^2 = \frac{1}{g_1^2} U^T R^T R U, $$ since $R$ is nonsingular integer and $R^T R$ is symmetric positive definite. So, no matter what, we have a way to find all primitive solutions $X$ with some $|X| \leq \mbox{bound}$ by taking $|u|, |v|$ up to some other bound we can figure out. $$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$ A more interesting alternative: for each divisor of $\det R,$ we may rewrite the eventual primitive solution with that gcd as a new recipe, $R_1 U$ for a new integer matrix $R_1$ that also solves $R_1^T G R_1 = n H.$
The example I like to show is solving $$ 2(x^2 + y^2 + z^2) - 113(yz + zx + xy)=0, $$ four "recipes," $$ \left( \begin{array}{r} x \\ y \\ z \end{array} \right) = \left( \begin{array}{r} 37 u^2 + 51 uv + 8 v^2 \\ 8 u^2 -35 uv -6 v^2 \\ -6 u^2 + 23 uv + 37 v^2 \end{array} \right) $$
$$ \left( \begin{array}{r} x \\ y \\ z \end{array} \right) = \left( \begin{array}{r} 32 u^2 + 61 uv + 18 v^2 \\ 18 u^2 -25 uv -11 v^2 \\ -11 u^2 + 3 uv + 32 v^2 \end{array} \right) $$
$$ \left( \begin{array}{r} x \\ y \\ z \end{array} \right) = \left( \begin{array}{r} 38 u^2 + 45 uv + 4 v^2 \\ 4 u^2 -37 uv -3 v^2 \\ -3 u^2 + 31 uv + 38 v^2 \end{array} \right) $$
$$ \left( \begin{array}{r} x \\ y \\ z \end{array} \right) = \left( \begin{array}{r} 29 u^2 + 63 uv + 22 v^2 \\ 22 u^2 -19 uv -12 v^2 \\ -12 u^2 -5 uv + 29 v^2 \end{array} \right) $$
In all four cases we simply discard occurrences when the resulting $x,y,z$ have a common factor. With the understanding that we negate all $x,y,z$ so that the entry with largest absolute value is positive, then sort so that $$ x \geq |y| \geq |z|, $$ here are the answers with maximum up to $1200$
jagy@phobeusjunior:~$ ./isotropy_binaries_combined 2 113 1200 | sort -n
x y z first line u v
29 22 -12 < 29, 63, 22 > 1 0
32 18 -11 < 32, 61, 18 > 1 0
37 8 -6 < 37, 51, 8 > 1 0
38 4 -3 < 38, 45, 4 > 1 0
188 171 -86 < 37, 51, 8 > 1 2
211 144 -82 < 38, 45, 4 > 1 2
226 123 -76 < 32, 61, 18 > 1 2
243 94 -64 < 29, 63, 22 > 1 2
246 88 -61 < 38, 45, 4 > 2 1
258 59 -44 < 37, 51, 8 > 2 1
264 38 -29 < 29, 63, 22 > 2 1
268 11 -6 < 32, 61, 18 > 2 1
396 262 -151 < 37, 51, 8 > 1 3
432 209 -134 < 38, 45, 4 > 1 3
472 129 -94 < 29, 63, 22 > 3 1
489 76 -58 < 32, 61, 18 > 3 1
516 458 -233 < 38, 45, 4 > 2 3
526 447 -232 < 37, 51, 8 > 2 3
628 311 -198 < 38, 45, 4 > 3 2
656 262 -177 < 32, 61, 18 > 2 3
671 232 -162 < 37, 51, 8 > 3 2
692 183 -134 < 29, 63, 22 > 2 3
726 47 -32 < 32, 61, 18 > 3 2
727 36 -22 < 29, 63, 22 > 3 2
804 787 -382 < 32, 61, 18 > 1 5
894 688 -373 < 29, 63, 22 > 1 5
953 946 -456 < 38, 45, 4 > 3 4
1034 492 -317 < 37, 51, 8 > 1 5
1062 443 -296 < 29, 63, 22 > 5 1
1102 363 -256 < 38, 45, 4 > 1 5
1123 314 -228 < 32, 61, 18 > 5 1
1159 1046 -528 < 32, 61, 18 > 1 6
1179 118 -88 < 38, 45, 4 > 5 1
1188 19 2 < 37, 51, 8 > 5 1
1199 1002 -524 < 29, 63, 22 > 1 6
x y z first line u v
I should probably point out that, while it was quite easy (after writing the C++ programs) to make a list of primitive solutions to $ 2(x^2 + y^2 + z^2) - 113(yz + zx + xy)=0 $ with $x \geq |y| \geq |z|$ for $x \leq 1200,$ and just as easy to identify the four square matrices $R_1,R_2,R_3,R_4$ used above, it was quite a big job to prove that these really do give all (ordered) primitive solutions. I have a pdf of the whole business in detail, about twenty pages Latex. Oh: in the above, we may always take $u,v \geq 0.$ It is a reasonable conjecture that the problem $ A(x^2 + y^2 + z^2) - B(yz + zx + xy)=0, $ with $\gcd(A,B)=0,$ $B > A > 0,$ and both $B-A$ and $B + 2A$ expressible in integers as $s^2 + 3 t^2,$ always works out with a finite number of such $R_i.$ No proof.
For the equation.
$$aX^2+bY^2+cZ^2=dXY+eXZ+fYZ$$
If you know any solution $(x,y,z)$ of this equation. Then the formula for the solution of the equation can be written immediately.
$$X=(dy+ez-ax)p^2+(fz-2by)ps+bxs^2+cxt^2+(fy-2cz)pt-fxst$$
$$Y=ayp^2+(ez-2ax)ps+(dx+fz-by)s^2+cyt^2-eypt+(ex-2cz)st$$
$$Z=azp^2-dzps+bzs^2+(ex+fy-cz)t^2+(dy-2ax)pt+(dx-2by)st$$
$p,s,t - $ any integers.
This thing surprised me, but perhaps it should not have. Given any quadratic form $g(x) = x' G x,$ where $G$ is a symmetric matrix, $x$ a column vector, and $x'$ its transpose, a row vector, we are taking $x$ and a new vector $v$ and creating $$ r = (v'Gv)x - 2 (x'Gv)v. $$ This gives us $$ r' G r = (v'Gv)^2 (x'Gx) - 4 (v'Gv)(x'Gv)^2+ 4 (x'Gv)^2 (v'Gv) = (x'Gx)(v'Gv)^2, $$ so $$ r' G r = (x'Gx)(v'Gv)^2. $$ This is Desboves, $$ g(r) = g(x) \left( g(v) \right)^2 . $$ In particular, if $g(x) = 0,$ then $g(r) = 0.$
Why do I call this stereographic projection? We start with an integer vector $x,$ and suppose that $g(x) = 0.$ Take a second integer vector $v.$ With a rational parameter $t,$ any rational vector can be written as $x+tv.$ Furthermore, $g(x+tv) = g(x) + 2 t (x'Gv) + t^2 g(v).$ Under the supposition that $g(x) = 0,$ we find the other null vector along this ray by setting $2 t (x'Gv) + t^2 g(v)=0,$ demanding $t \neq 0,$ dividing out by $t,$ then solving, which gives $t g(v) = -2 (x'Gv).$ So the rational $$ t = -2 (x'Gv)/ g(v), $$ and $$ x+tv = \frac{xg(v) - 2 (x'Gv)v}{g(v)}. $$ When $g(x) = 0,$ this $g(x+tv) =0$ gives the other rational point along that line and on the surface $g=0.$ It may not be composed of integers, though, so we take the integer multiple $$ r = x g(v) - 2 (x'Gv)v. $$ We do get $g(r) = 0,$ the only problem is that the gcd of the entries of $r$ is not entirely predictable. If we agree to divide out by the gcd, we are guaranteed to get all integer primitive solutions to $g=0$ this way.
some detail on individ's answer, based on a new comment by Tito a few hours ago. individ wrote with some minus signs on the mixed terms, in the pari output below I show how to derive the expressions and I confirm the Desboves type identity
$$ aX^2 + b Y^2 + c Z^2 - d XY - e XZ - f YZ = (ax^2 + b y^2 + c z^2 - d xy - e xz - f yz) \left(ap^2 + b s^2 + c t^2 - d ps - e pt - f st \right)^2 $$
with $X,Y,Z$ as given in his answer. I checked this in gp-pari, it is stereographic projection around a known solution, which has lower case $x,y,z.$ When finished, there is the Desboves type identity abaove, and this holds even when $(x,y,z)$ is not really a solution..
let: X = x + u * p; Y = y + u * s; Z = z + u * t
let: poly = a * X^2 + b * Y^2 + c * Z^2 - d * X * Y - e * X * Z - f * Y * Z
parisize = 4000000, primelimit = 500509
? X = x + u * p; Y = y + u * s; Z = z + u * t
%1 = t*u + z
? poly = a * X^2 + b * Y^2 + c * Z^2 - d * X * Y - e * X * Z - f * Y * Z
%2 = a*x^2 + ((2*a*p + (-d*s - e*t))*u + (-d*y - e*z))*x + ((a*p^2 + (-d*s - e*t)*p + (b*s^2 - f*t*s + c*t^2))*u^2 + ((-d*y - e*z)*p + ((2*b*s - f*t)*y + (-f*z*s + 2*c*t*z)))*u + (b*y^2 - f*z*y + c*z^2))
?
?
? poly0 = polcoeff(poly, 0, u)
%3 = a*x^2 + (-d*y - e*z)*x + (b*y^2 - f*z*y + c*z^2)
?
? poly1 = polcoeff(poly, 1, u)
%4 = (2*a*p + (-d*s - e*t))*x + ((-d*y - e*z)*p + ((2*b*s - f*t)*y + (-f*z*s + 2*c*t*z)))
?
? poly2 = polcoeff(poly, 2, u)
%5 = a*p^2 + (-d*s - e*t)*p + (b*s^2 - f*t*s + c*t^2)
?
?
X,Y,Z become:
p,s,t
denom = a * p^2 + b * s^2 + c * t^2 - d * p * s - e * p * t - f * s * t
? denom - poly2
%7 = 0
?
?
? XX = x * poly2 - p * poly1
%8 = (-a*p^2 + (b*s^2 - f*t*s + c*t^2))*x + ((d*y + e*z)*p^2 + ((-2*b*s + f*t)*y + (f*z*s - 2*c*t*z))*p)
?
? YY = y * poly2 - s * poly1
%9 = (-2*a*s*p + (d*s^2 + e*t*s))*x + (a*y*p^2 + (-e*t*y + e*z*s)*p + ((-b*s^2 + c*t^2)*y + (f*z*s^2 - 2*c*t*z*s)))
?
? ZZ = z * poly2 - t * poly1
%10 = (-2*a*t*p + (d*t*s + e*t^2))*x + (a*z*p^2 + (d*t*y - d*z*s)*p + ((-2*b*t*s + f*t^2)*y + (b*z*s^2 - c*t^2*z)))
?
? polypoly = a * XX^2 + b * YY^2 + c * ZZ^2 - d * XX * YY - e * XX * ZZ - f * YY * ZZ
polypoly - poly0 * poly2^2
?
?
? polypoly - poly0 * poly2^2
%12 = 0
?
?
? WOW