$\newcommand{\eps}{\varepsilon}$ $\newcommand{\rad}{\mathrm{rad}}$

At least, under the abc conjecture, there can be only finitely many pairs $(a,b)$ with $b>a>1$ coprime such that $a^b+b^a$ is a square.

As a reminder, the conjecture says that to any $\eps>0$ there corresponds some $K_\eps>0$ such that whenever $u,v$, and $w$ are coprime positive integers with $u+v=w$, one has $\rad(uvw)>K_\eps w^{1-\eps}$. Here $\rad(z)$ is the product of all primes dividing $z$ (thus, for instance, $\rad(8)=2$, $\rad(9)=3$, $\rad(10)=10$, $\rad(11)=11$, and $\rad(12)=6$).

Suppose now that $a^b+b^a=c^2$ with coprime integers $b>a\ge 3$ and $c>0$ (the case $a=2$ is resolved above). Applying the abc conjecture with $u=a^b$, $v=b^a$, $w=c^2$, and $\eps=0.05$, and making the key observation $\rad(a^bb^ac^2)\le abc$, we conclude that $$ Kc^{2\cdot 0.95} < abc $$ with an absolute constant $K>0$. At the same time, we have $c^2>a^b$ and $c^2>b^a$, implying $a<c^{2/b}$ and $b<c^{2/a}$, respectively. Consequently, $$ Kc^{1.9} < c^{(2/b)+(2/a)+1}, $$ showing that either $\frac1b+\frac1a>0.4$, or $Kc^{0.1}<1$. Clearly, there are only finitely many values of $c$ satisfying the latter condition, and to each value corresponds finitely many pairs $(a,b)$. On the other hand, since $\frac1b+\frac13\ge\frac1b+\frac1a>0.4$ implies $b<15$, there are only finitely many pairs $(a,b)$ satisfying the former condition. Thus, the total number of exceptional pairs $(a,b)$ is also finite.


(21-03-2020) Update: As no unconditional answer has yet been given, I will include a few more conditions that any solution must satisfy. The results used are rather advanced, so I will only include references, not proofs.

Lemma 5: Let $a$ and $b$ be positive integers such that $a^b+b^a$ is a perfect square. Then $\gcd(a,b)\leq3$.

Proof. Such a pair of positive integers yields a nontrivial integral solution to $$x^d+y^d=z^2,\tag{3}$$ where $d:=\gcd(a,b)$. This implies $d\leq3$ by this paper${}^1$.$\qquad\square$

Proposition 6: Let $a$ and $b$ be positive integers such that $a^b+b^a$ is a perfect square. Then $\gcd(a,b)\leq2$.

Proof. By the preceding lemma it suffices to show that $\gcd(a,b)=3$ is impossible. If $\gcd(a,b)=3$ then $a^{\tfrac b3}$ and $b^{\tfrac a3}$ are integers satisfying $$\Big(a^{\tfrac b3}\Big)^3+\Big(b^{\tfrac a3}\Big)^3=z^2,$$ for some integer $z$, which means that, after swapping $a$ and $b$ if necessary, either $$a^{\tfrac b3}=\frac{x(x^3-8y^3)}{w^2}z^2 \qquad\text{ and }\qquad b^{\tfrac a3}=\frac{4y(x^3+y^3)}{w^2}z^2,$$ for integers $x$, $y$ and $z$ with $x$ odd and $x$ and $y$ coprime, and $w:=\gcd(3,x+y)$, or $$a^{\tfrac b3}=\frac{x^4+6x^2y^2-3y^4}{w^2}z^2 \qquad\text{ and }\qquad b^{\tfrac a3}=\frac{3y^4+6x^2y^2-x^4}{w^2}z^2,$$ for $x$, $y$ and $z$ with $x$ and $y$ coprime and $x$ coprime to $3$, and $w=\gcd(2,x+1,y+1)$. These complete parametrizations of the integral solutions to $(3)$ when $d=3$ are taken from section 7.2 of this article${}^2$.

For the second parametrization, because $x$ is coprime to $3$ we see that $$\nu_3\Big(a^{\tfrac b3}\Big)=\nu_3(z^2) \qquad\text{ and }\qquad \nu_3\Big(b^{\tfrac a3}\Big)=\nu_3(z^2),$$ where $\nu_p(t)$ denotes largest integer $k$ such that $p^k$ divides $t$. It follows that $a\nu_3(b)=b\nu_3(a)$, and from $\gcd(a,b)=3$ it follows that either $\nu_3(a)=1$ or $\nu_3(b)=1$, so either $a$ divides $b$ or $b$ divides $a$, respectively. This means either $a=3$ or $b=3$.

If $a=3$ then the identity $$3^{\tfrac b3}=a^{\tfrac b3}=\frac{x^4+6x^2y^2-3y^4}{w^2}z^2,$$ shows that $z^2=3^{\tfrac b3}$. The parametrization for $b^{\tfrac a3}$ then shows that $3^{\tfrac b3}$ divides $b^{\tfrac a3}=b$, which quickly implies $b=3$. But $3^3+3^3=54$ is not a perfect square; a contradiction. If $b=3$ a similar argument shows that then $a=3$, and this shows that the second parametrization yields no solutions to our original problem.

For the first parametrization, note that $b$ is even and hence $a^{\tfrac b3}$ is a perfect square, hence so is $$w^2z^{-2}a^{\tfrac b3}=x(x^3-8y^3)=x^4-8xy^3.$$ This means there is some integer $c$ such that $x^4-8xy^3=(x^2-2c)^2$ and so $$cx^2+2y^3x-c^2=0.$$ In particular the discriminant $\Delta$ of this quadratic polynomial in $x$ is a perfect square, say $\Delta=(2e)^2$, where $$4e^2=\Delta=(2y^3)^2-4c(-c)^2=4(y^6+c^3).$$ It is a classical result that then for some nonnegative integer $k$ we have $$(|e|,c,|y|)=(3k^3,2k^2,k).$$ Plugging this in and solving the quadratic equation for $x$ yields $x\in\{\pm k,\pm2k\}$ where $y=\pm k$. Because $x$ and $y$ are coprime it follows that $k=1$, so $y=\pm1$ and $x\in\{\pm1,\pm2\}$. Then for $a^{\tfrac b3}$ to be a perfect square we must have $\{x,y\}=\{1,-1\}$, but then $b=0$, a contradiction. This shows that the second parametrization also yields no solutions to our original problem. In conclusion $\gcd(a,b)=3$ is impossible. $\quad\square$.

A small step towards a complete proof of the original problem would be to show that any solution other than $\{a,b\}=\{2,6\}$ must have $a$ and $b$ coprime, which seems likely. So for now, I propose the following conjecture:

Conjecture 7: Let $a$ and $b$ be positive integers such that $a^b+b^a$ is a perfect square and $\gcd(a,b)=2$. Then $\{a,b\}=\{2,6\}$.

Of course such solutions yield Pythagorean triples, for which the parametrization is well known. Perhaps arguments similar to those of Proposition 6 can be used here. I hope to give another update resolving this conjecture soon.


References

  1. H. Darmon and L. Merel, Winding quotients and some variants of Fermat’s last theorem, J.Reine Angew. Math. 490 (1997), 81–100.
  2. H. Darmon, A. Granville, On the equations $x^p+y^q=z^r$ and $z^m=f(x, y)$, Bulletin of the London Math. Society, no 129, 27 part 6, November 1995, pp. 513–544.

Original answer:

I'll collect a few partial results here. Let $a$, $b$ and $c$ be positive integers with $a,b>1$ such that $$a^b+b^a=c^2,$$ and let $d=\gcd(a,b)$. First two lemmas that are repeatedly useful.

Lemma 1: If $m$ and $n$ are positive integers with $m>n$ and not both even, such that $m+n$ and $m-n$ are both powers of $2$, then $m=2^k+1$ and $n=2^k-1$ for some positive integer $k$.

Proof. If $m+n=2^u$ and $m-n=2^v$ then $$m=\frac{(m+n)+(m-n)}2=\frac{2^u+2^v}2=2^{v-1}(2^{u-v}+1),$$ $$n=\frac{(m+n)-(m-n)}2=\frac{2^u-2^v}2=2^{v-1}(2^{u-v}-1),$$ and hence $v=1$ because one of $m$ and $n$ is odd. Then $k=u-v$.$\qquad\square$

Lemma 2: The only perfect power that is one less than a square is $8$.

Proof. There are fairly elementary proofs, but it also follows from Mihailescu’s theorem.$\qquad\square$

Proposition 3: If $a$ is a power of $2$ then $(a,b)=(2,6)$.

Most of this was proved in the original question by TheSimpliFire and Haran.

Proof. Let $a=2^e$. If $b$ is odd then writing $$(c-b^{2^{e-1}})(c+b^{2^{e-1}})=c^2-b^a=a^b=2^{be},$$ shows that both factors on the left hand side are powers of $2$. Then by Lemma 1 we have $c=2^v+1$ and $$b^{2^{e-1}}=2^v-1,$$ for some positive integer $v$ because $b$ is odd. Hence by Lemma 2 either $v=1$ or $2^{e-1}=1$. Clearly $v=1$ is impossible, so $2^{e-1}=1$ and so $e=1$. Then comparing exponents shows that $b=v+2$ and so $$v+2=b=2^v-1,$$ which is easily seen to have no integral solutions. Hence $b$ is even, say $b=2f$. Then we have the following Pythagorean triple: $$c^2=a^b+b^a=(2^e)^{2f}+(2f)^{2^e}=(2^{ef})^2+((2f)^{2^{e-1}})^2.$$ Then there exist positive integers $k$, $m$ and $n$ with $m>n$ and $\gcd(m,n)=1$ such that either $$c=k(m^2+n^2),\qquad2^{ef}=k(m^2-n^2),\qquad (2f)^{2^{e-1}}=2kmn,\tag{1}$$ $$\text{or}$$ $$c=k(m^2+n^2),\qquad2^{ef}=2kmn,\qquad (2f)^{2^{e-1}}=k(m^2-n^2).\tag{2}$$ In case the triple is of the form $(2)$, the middle identity shows that $k$, $m$ and $n$ are all powers of $2$, so in particular $n=1$ because $m$ and $n$ are coprime and $m>n$. Then the latter identity shows that $$(2f)^{2^{e-1}}=k(m^2-1)=k(m-1)(m+1),$$ where the factors $m-1$ and $m+1$ are odd and $k$ is a power of $2$, so both $m-1$ and $m+1$ are $2^{e-1}$-th powers. But for $e>1$ no two $2^{e-1}$-th powers of positive numbers differ by $2$, so $e=1$. Writing $k=2^u$ and $m=2^v$ we see that $u+v+1=f$, where $v\geq1$ because $m>n$. By comparing powers in the above we find that $$u+v+1=2^{u-1}(2^{2v}-1)=2^{u-1}(2^v-1)(2^v+1).$$ Of course $2^v+1>2$, so $2^{u-1}=1$ as otherwise $$2^{u-1}(2^v+1)>2^{u-1}+2^v+1\geq u+v+1,$$ a contradiction. Hence $u=1$ and $2^{2v}-1=v+2$, so also $v=1$. This yields the solution $(a,b)=(2,6)$.

On the other hand, if the Pythagorean triple is of the form $(1)$ then $k$, $m-n$ and $m+n$ are powers of $2$ because $$2^{ef}=k(m^2-n^2)=k(m-n)(m+n).$$ Because $m$ and $n$ are not both even, by Lemma 1 there exists a positive integer $v$ such that $m=2^v+1$ and $n=2^v-1$, and so the above shows that $k=2^{ef-v-2}$. Plugging this into the other equation yields $$(2f)^{2^{e-1}}=2kmn=2^{ef-v-1}(2^{2v}-1),$$ and writing $f=2^wg$ with $g$ odd then implies $$g^{2^{e-1}}=2^{2v}-1,$$ which by Lemma 2 implies that $2^{e-1}=1$, and hence $e=1$. Then $f=kmn$ and if $kmn\geq4$ then $$2^{kmn}\geq(kmn)^2\geq km^2>k(m^2-n^2),$$ so $f=kmn\leq3$. Then $b\leq6$ and clearly $b=2$ and $b=4$ do not yield solutions.$\qquad\square$

Proposition 4: If $d$ is even then $d=2$.

Proof. Suppose $d=2e$ and let $a=2eA$ and $b=2eB$. Then $e$ is odd as otherwise $c^2$ is the sum of two fourth powers, which is well known to be impossible by a classical result by Fermat. Now $$c^2=a^b+b^a=(a^{eB})^2+(b^{eA})^2,$$ is a pythagorean triple and hence there exist positive integers $k$, $m$ and $n$ with $m>n$ and $\gcd(m,n)=1$ such that $$a^{eB}=k(m^2-n^2),\qquad b^{eA}=2kmn,\qquad c=k(m^2+n^2),$$ where we can exchange the roles of $a$ and $b$ if necessary. It is clear that $$k=\gcd(a^{eB},b^{eA})=\gcd((dA)^B,(dB)^A)^e=d^{e\ell},$$ where $\ell\geq\min\{A,B\}$. In particular $k$ is an $e$-th power, and hence the factorizations $$(a^B)^e=k(m^2-n^2)=k(m-n)(m+n) \qquad\text{ and }\qquad (b^A)^e=2kmn,$$ show that, up to powers of $2$, the factors $m$, $n$, $m+n$ and $m-n$ are also $e$-th powers. That is to say, $$m=2^tp^e,\qquad n=2^uq^e,\qquad m+n=2^vr^e,\qquad m-n=2^ws^e,$$ for odd positive integers $p$, $q$, $r$ and $s$, and nonnegative integers $t$, $u$, $v$ and $w$, and $t+u+1\equiv v+w\equiv0\pmod{e}$. Then $$m=\frac{(m+n)+(m-n)}{2}=\frac{2^vr^e+2^ws^e}{2}=2^{v-1}r^e+2^{w-1}s^e,$$ $$n=\frac{(m+n)-(m-n)}{2}=\frac{2^vr^e-2^ws^e}{2}=2^{v-1}r^e-2^{w-1}s^e,$$ and at least one of $m$ and $n$ is odd, so either $v=1$ or $w=1$ (but not both) or $v=w=0$.

If either $v=1$ or $w=1$ (but not both) then $m$ and $n$ are both odd, so $t=u=0$ and hence $e=1$.

If $v=w=0$ then still either $t=0$ or $u=0$ because $m$ or $n$ is odd. If $u=0$ then $e\mid t+1$ and $$2^{t+1}p^e=2m=(m+n)+(m-n)=r^e+s^e,$$ and so it follows from Fermats last theorem that $e=1$. The same holds if $t=0$.$\qquad\square$