$(a+b)^2+4ab$ and $a^2+b^2$ are both squares
Solution 1:
Let $q^2 = (a+b)^2$, $p^2 = a^2 + b^2$ and $r^2 = (a+b)^2 + 4ab$. Note that $r^2 + p^2 = 2q^2$. Now let $X = r+p$, $Y = r - p$ and $Z = 2q$.
You can easily show that $r^2 + p^2 = 2q^2$ if and only if $X^2 + Y^2 = Z^2$.
So, each solution can be found as follows:
- Pick a Pythagorean triple $(X, Y, Z)$ in which $Z$ is even.
- Let $q = \frac{Z}{2}$, $r = \frac{X+Y}{2}$ and $p = \frac{X-Y}{2}$.
- Check if there is any Pythagorean triple $(a', b', p)$ for all $a', b'$ that $a'+b' = q$.
Hope this helps :)
Solution 2:
In general, to make two quadratic polynomials as squares like,
$$a^2+b^2 = c^2\tag1$$
$$(a+b)^2+4ab = d^2\tag2$$
will yield one quartic to be made a square. Given the complete solution to Pythagorean triples as $a,b = m(x^2-y^2),\,2mxy$, then,
$$m^2(x^4 + 12 x^3 y + 2 x^2 y^2 - 12 x y^3 + y^4) = d^2$$
We can suppress $m,y$ without loss of generality,
$$u^4 + 12 u^3 + 2 u^2 - 12 u+1=v^2\tag3$$
Since you've found rational points to $(3)$ like $u =x/y = 3/2,\; 5/1,\; 7/85,\dots$, then it is birationally equivalent to an elliptic curve, so there is an infinite more (like $u = 616264191/46041814$). This yields,
$$a,b =377661704472473885,\;56747842513764948$$
Thus your system $(1)$, $(2)$ has an infinite number of primitive integer solutions, all of which are rational points on $(3)$.
P.S. For an elementary discussion on how to find more points on $(3)$, see this post.