Find all positive integers $x,y$ such that $(x+y)(xy+1)$ is a power of $2$ [duplicate]

You have a good start in using uniqueness of factorization.

Because $xy+1\ge x+y$, we have $b\ge a$. Therefore $xy\equiv-1\pmod{2^a}$. As we also have $x\equiv-y\pmod{2^a}$, we can conclude that $$ x^2\equiv-xy\equiv1\pmod{2^a}. $$

Assume that $a\ge3$. Then we know that the group $\mathbb{Z}_{2^a}^{\times}$ of units of this residue class ring is isomorphic to $C_2\times C_{2^{a-2}}$. Therefore there are exactly four elements of order two in this group, and they are easily seen to be the residue classes of $\pm1$ and $\pm1+2^{a-1}$. The same reasoning applies to $y$. As we know that $0<x,y<2^a=x+y$, the remaining possibilities are $\{x,y\}=\{1,2^a-1\}$ and $\{x,y\}=\{2^{a-1}-1,2^{a-1}+1\}$. Both of these work, the first choice leads to $b=a$ the latter to $b=2a-2$.

I leave the cases $a<3$ to you.

Hint: Your idea will work. The discriminant is $2^{2a} -4(2^b-1)$. For simplicity divide by $4$. We get $2^{2c}-4\cdot 2^b+1$. Now an analysis of size will bring the problem down to small cases. A useful fact is that there is no perfect square strictly between $x^2$ and $(x+1)^2$.