Prove that $a < b\sqrt{3}$ under conditions given
If $a$ and $b$ share a common factor, then it is shared also by $a+b$ and $ab$, and therefore not by $ab+1$ and we could not have item 2). Therefore $a$ and $b$ are relatively prime.
Item 2) implies that $b^2-1$ is divisible by both $a+b$ and $a-b$, and hence their least common multiple $\mathrm{lcm}(a+b,a-b) = (a+b)(a-b) / \mathrm{gcd}(a+b,a-b)$.
Since $b^2-1 > 0$, it follows that $b^2-1 \geq \mathrm{lcm}(a+b,a-b) = (a+b)(a-b) / \mathrm{gcd}(a+b,a-b)$.
What common factors can $a+b$ and $a-b$ have? If they share an odd prime $p$ as a common factor, then $a$ and $b$ share that prime as a common factor; but $a$ and $b$ are relatively prime by the first entry on this list. If $a+b$ and $a-b$ are both divisible by $4$, then $a$ and $b$ are both even, which we also ruled out. Thus $\mathrm{gcd}(a+b,a-b)$ is either $1$ or $2$.
In summary, $b^2 - 1 \geq (a+b)(a-b)/2$. Multiply through by $2$ to get $2b^2 - 2 \geq a^2 - b^2$. Add $b^2$ for $3b^2 - 2 \geq a^2$, from which it follows that $3b^2 > a^2$. Then take the square root.
Edit: It was requested in the comments that I explain step 2. To wit:
2+. Since $ab+1$ is divisible by $a+b$, so is $(ab+1) - b(a+b) = -b^2 + 1 = -(b^2-1)$.
2-. Since $ab-1$ is divisible by $a-b$, so is $(ab-1) - b(a-b) = b^2-1$.
It is perhaps useful to mention some words about how I found the solution. The problem asked to prove the bound $a < b\sqrt 3$. I asked myself: rather than prove that specific bound, can we give some bound? I.e. can we disprove the possibility that $a \gg b$?
Well, I said, if $a \gg b$, then $a \pm b \approx a$. So I know that $ab \pm 1$ is divisible by something approximately $a$. Well, what must the quotient be? $ab \pm 1 \approx ab$, so $(ab \pm 1) / (a \pm b) \approx b$. Ok, so let's subtract off $b(a\pm b)$ and see what's left.
Great, then $b^2 - 1$ is divisible by something approximately $a$. Since $b>1$, this already gives some bound: it says that $b^2 \gtrsim a$. I wanted a bound of the form $b \gtrsim a$, up to some fixed constant, and I'm not there yet, but I've at least ruled out, say, $a \sim b^{1000}$.
Ok, I haven't used both conditions from line 2). I know that something approximately $b^2$ is divisible by two different things that are approximately $a$. I want $b^2 \gtrsim a^2$ --- since I have two terms, I have $b^2 \gtrsim \mathrm{lcm}(\approx a,\approx a)$. And lcms are usually products, because large numbers tend to have few large common factors.
But can I prove that? Ah, I haven't used the stated conditions from 2), just their corollaries about $b^2-1$. Those stated corollaries obviously imply that $a,b$ are linearly independent. And that's almost what I want! Indeed, I want to find a bound on $\mathrm{gcd}(a+b,a-b)$, and I have a bound on $\mathrm{gcd}(a,b)$ (in fact, I know the latter is $1$). So perhaps that's all I need? Yes: $\mathrm{gcd}(a,b)=1$ implies $\mathrm{gcd}(a+b,a-b)\leq 2$.