Determine all $a,b,c\in \mathbb{Z}$ for which the equation $(a^2+b^2)x^2-2(b^2+c^2)x-(c^2+a^2)=0$ has rational roots.

This is partial solution.

As Χpẘ suggested, we need to determine $a,b,c$ which makes $a^4+b^4+c^4+a^2b^2+a^2c^2+3b^2c^2$ perfect square. We know that $(a^2+b^2+c^2)^2=a^4+b^4+c^4+2a^2b^2+2b^2c^2+2c^2a^2$ is a square.

We know that $(a,0,0), (0,b,0), (0,0,c)$ is solution. Therefore, let's assume that at most one of $a,b,c$ is $0$. Also, we are dealing with squares, so we can assume that $a,b,c$ is all non-negative. Since if $(x,y,z)$ is a solution, then $(kx,ky,kz)$ is also a solution, we may assume $\gcd(a,b,c)=1$. Also, the equation is symmetric with respect to $b$ and $c$, so we may assume that $b \ge c$.

Case 1: $a^4+b^4+c^4+2a^2b^2+2b^2c^2+2c^2a^2 = a^4+b^4+c^4+a^2b^2+a^2c^2+3b^2c^2$

If then, $(b^2-a^2)(c^2-a^2)=a^4$. If $a=0$, there is no solution since $b,c$ is nonzero. Similarly, we cannot assume that $b$ or $c$ is zero.

Also if $c \le b < a$, then $0>b^2-a^2>-a^2$ and $0>c^2-a^2>-a^2$, so no solution. If $c \le a \le b$, then LHS is $0$ or negative. Therefore $a<c\le b$.

Let $b^2-a^2=\frac{p}{q}a^2$ and $c^2-a^2=\frac{q}{p}a^2$. Clearly $p \ne q$ and from $b \ge c$, $p > q$. Then we know that both $\frac{p+q}{q}$ and $\frac{p+q}{p}$ are perfect squares, also their reciprocals $\frac{q}{p+q}$ and $\frac{p}{p+q}$. Their sum is exactly $1$ and we know all nontrivial rational solutions to this equation corresponds to Pythagorean triple. Therefore, take any Pythagorean triple $l,m,n$ with $l^2+m^2=n^2$ and $l<m<n$, we have $b^2l^2=a^2n^2$ and $c^2m^2=a^2n^2$. It follows that $a=lm$, $b=mn$, $c=ln$.

Case 2: $a^4+b^4+c^4+2a^2b^2+2b^2c^2+2c^2a^2 \ne a^4+b^4+c^4+a^2b^2+a^2c^2+3b^2c^2$

This case is really hard... I cannot make any progress worth mentioning.

I made a computer search for the solutions under 1k of Case 2. There are about 760 solutions, and you can see the list and code used here.

def is_square(apositiveint):
    x = apositiveint // 2
    seen = set([x])
    while x * x != apositiveint:
        x = (x + (apositiveint // x)) // 2
        if x in seen: return False
        seen.add(x)
    return True

for i in range(0, 1000):
    for j in range(0, 1000):
        for k in range(j, 1000):
            number = i ** 4 + j ** 4 + k ** 4 + 3 * j * j * k * k + i * i * (j * j + k * k)
            if number > 2 and not (i == j == 0 or j == k == 0 or k == i == 0) and is_square(number) and (i * i + j * j + k * k) ** 2 != number:
                print i, j, k, (number ** 0.5 - (i * i + j * j + k * k)) / 18

https://docs.google.com/spreadsheets/d/1gij-LfwLYKCq5TnYlRtDnbem988Udu6zZU-a7iGMjtY/edit?usp=sharing