proving that $ab$ is a perfect square.

Solution 1:

Since $[a^{3n}-1][b^{3n}-1]$ is a square and $[a^{n}-1][b^{n}-1]$ is a non-zero square, the quotient $[a^{2n}+a^n+1][b^{2n}+b^n+1]$ is also a square for all $n\in\mathbb N$.

Suppose that $ab$ is not a square. A consequence of quadratic reciprocity is that there are infinitely many primes $p$ such that $\left(\frac{ab}p\right) = \left(\frac{3}p\right)=-1$ (the density of such primes is at least $\tfrac14$). Fix $p$ to be one such prime.

Now, exactly one of $a$ and $b$ is a quadratic residue mod $p$. Choosing $n=(p-1)/2$, Euler's criterion shows that $a^{2n}+a^n+1$ is $3 \pmod p$ if $a$ is a residue and $1\pmod p$ if $a$ is a non-residue. Therefore $[a^{2n}+a^n+1][b^{2n}+b^n+1] \equiv 3 \pmod p$, so it can't be a square.

This is a very nice problem, and I imagine that other quite different attacks are possible. I'd like to know if there is a way to deduce the much stronger statement that $a = b$, since that seems like the only plausible way to make $[a^{n}-1][b^{n}-1]$ always a non-zero square.