What is the minimum value of $a+b+\frac{1}{ab}$ if $a^2 + b^2 = 1$?

For the case when $a,b>0,$ I used AM-GM Inequality as follows that:

$\frac{(a+b+\frac{1}{ab})}{3} \geq (ab\frac{1}{ab})^\frac{1}{3}$

This implies that $(a+b+\frac{1}{ab})\geq 3$. Hence, the minimum value of $(a+b+\frac{1}{ab})$ is 3

But the answer is $2+\sqrt{2}$ ... how is it ?

If you want to use AM-GM for this, you need to ensure that the equality condition can be met along with the constraint, by "balancing coefficients". Illustrated below:

$$a+b+\frac1{ab} = a+b +\frac{1}{2\sqrt2 ab}+\left(1-\frac1{2\sqrt2}\right)\frac1{ab}$$

Now by AM-GM, $$ a+b +\frac{1}{2\sqrt2 ab}\ge \frac{3}{\sqrt2}$$

and for the remaining term we have again $$\left(1-\frac1{2\sqrt2}\right)\frac1{ab} \ge \left(1-\frac1{2\sqrt2}\right)\frac{2}{a^2+b^2} = 2-\frac{1}{\sqrt2}$$

Combining these results, we have $a+b+\dfrac1{ab} \ge 2 + \sqrt2$, with equality iff $a=b=\frac1{\sqrt2}$.

Added: The key here is of course knowing how to split the LHS, which is by noting that if for equality we need $a=b=\dfrac1{k~ab}$ and for the constraint we need $a^2+b^2=1$, what could be the value of $k$. The rest is then easy applications of AM-GM.

$2ab = (a+b)^2 - (a^2+b^2)$, hence we need to find the minimum value of $a + b + \frac{2}{(a+b)^2 - (a^2+b^2)}$ $ = a + b + \frac{2}{(a+b)^2 - 1}$.

Let us find the maximum value (k) of $a+b$ given $a^2+b^2=1$. Thinking graphically, when $a+b = k$ is tangent to the circle, its gradient is $-1$ and hence the gradient of the normal is $1$, or $a=b$. Hence $k = \frac{1}{\sqrt2} + \frac{1}{\sqrt2} = \sqrt{2}$, which gives $\sqrt{2} + \sqrt{2} + \frac{2}{2-1} = 2 + \sqrt{2}$ as the answer.

This relies on the fact that $f(x) = x + \frac{2}{x^2 - 1}$ is decreasing for $x \in (1, \sqrt{2}]$. Partial fractions gives $f(x) = x + \frac{1}{x-1} - \frac{1}{x+1}$ and taking the derivative shows that $f'(x) < 0$ in the given interval. There might be a way without calculus.

The minimum does not exist. Try $a>0$ and $b\rightarrow 0^-$.

Let the minimum value of $a+b+\frac{1}{ab} = k$. Both curves must be tangent to each other, which means they must have the same gradient. Thus $a^2 + b^2 = 1 \implies \frac{db}{da} = -\frac{a}{b}$, and:

$$a+b+\frac{1}{ab} = k \implies 1 + \frac{db}{da} - \frac{1 + db/da}{(ab)^2} = 0$$

$$\implies (ab)^2 \left(1 - \frac{a}{b} \right) - 1 - -\frac{a}{b} = 0$$ $$\implies a^2b^2 - a^3b - 1 + \frac{a}{b} =0$$ $$\implies a^2b^3 - a^3b^2 - b + a = 0$$ $$\implies a^2b^2(b-a) - (b-a) =0$$ $$\implies (ab+1)(ab-1)(b-a) = 0$$

but only $b-a = 0$ intersects with $a^2 + b^2 = 1$. This gives $a = b = \frac{1}{\sqrt2}$, and hence $k = \frac{1}{\sqrt2} + \frac{1}{\sqrt2} + \frac{1}{1/2} = 2 + \sqrt{2}$.