$a,b>0$ then: $\frac{1}{a^2}+b^2\ge\sqrt{2(\frac{1}{a^2}+a^2)}(b-a+1)$

Due to @Calvin Lin, I will post solution based on his hint later.

Now I get solution by AM-GM. Notice that: $\frac{1}{a^2}+a^2=\left(\frac{1}{a}+a\right)^2-2=\left(\frac{1}{a}+a+\sqrt{2}\right)\left(\frac{1}{a}+a-\sqrt{2}\right)$

Then using AM-GM: $$2\frac{\sqrt{2}\frac{1}{a^2}}{\sqrt{\frac{1}{a^2}+a^2}}+\frac{\frac{1}{a}+a+\sqrt{2}}{2+\sqrt{2}}+\frac{\frac{1}{a}+a-\sqrt{2}}{2-\sqrt{2}}\ge\frac{4}{a}$$ Or:$$\frac{\sqrt{2}\frac{1}{a^2}}{\sqrt{\frac{1}{a^2}+a^2}}+a\ge\frac{1}{a}+1$$ Similarly, $$\frac{\sqrt{2}b^2}{\sqrt{\frac{1}{a^2}+a^2}}+a+\frac{1}{a}\ge2b+1$$ The rest is obvious: $$\frac{\frac{1}{a^2}+b^2}{\sqrt{\frac{1}{a^2}+a^2}}\ge\sqrt{2}(b-a+1)$$

Since OP posted their own version, I'd post my solution.

WTS $ b^2 - \sqrt{2 ( \frac{1}{a^2 } + a^2 )} \cdot b + [\frac{1}{a^2} + \sqrt{2 ( \frac{1}{a^2 } + a^2 )} \cdot (a-1) ] \geq 0$.

We will prove this by viewing it as a quadratic in $b$, and showing that the discriminant is $ \leq 0$ for $ a > 0$, hence the value is always $ \geq 0 $.

$$D = [ \sqrt{2 ( \frac{1}{a^2 } + a^2 )}] ^2 - 4[\frac{1}{a^2} + \sqrt{2 ( \frac{1}{a^2 } + a^2 )} \cdot (a-1) ] \\ = - \frac{ 2 ( a-1) ( a^3+a^2+a+1 - 2\sqrt{2} \sqrt{ a^4 + 1 }) }{a^2} .$$

Observe that $ ( a^3 + a^2 + a + 1 ) ^2 - 8 (a^4 + 1) = (a-1) ( a^5 + 3a^4 - 2a^3 + 2a^2 + 5a + 7) $.
Clearly the second term is positive for $ a > 0$.

Hence

• If $ a > 1 $, then $ a^3 + a^2 + a + 1 > \sqrt{ 8 (a^4 + 1)}$, so $ D < 0 $
• If $ a = 1$, then $ D = 0 $
• If $ 0 < a < 1$, then $a^3 + a^2 + a + 1 < \sqrt{ 8 (a^4 + 1)},$ so $ D < 0 $.

Equality holds when $ D = 0 \Rightarrow a = 1 $ with corresponding root $b = 1$.

Modification of OP's solution.

Lemma: $$\frac{ \sqrt{2} x^2 } { \sqrt{ \frac{1}{a^2} + a^2 } } + \frac{ a^2 - a + 1 } { a } \geq 2 x . $$

Proof: We apply AM-GM directly. It remains to show that

$$ \sqrt{ \frac{ 2a^2} { a^4 +1 }} \times { \frac{ a^2 - a + 1 } { a } } \geq 1$$

or that $2 ( a^2 - a + 1 ) ^2 \geq (a^4 + 1) $, which is true since $2 ( a^2 - a + 1 ) ^2 - (a^4 + 1) = (a-1)^4$.
Equality holds when $ a = 1, x = 1 $.

Corollary: Apply the lemma to $ x = \frac{1}{a}$ and $x = b$, sum them up and shift terms around to get the desired inequality.
Equality holds when $ a = 1, b = 1 $.

Notes

• (I thought that) My contribution here is to simplify the AM-GM that OP did.
  • Since $ \frac{\frac{1}{a} + a +\sqrt{2} } { 2 + \sqrt{2} } + \frac{ \frac{ 1}{a} + a - \sqrt{2} } { 2 - \sqrt{2} } = 2\frac{ a^2 - a + 1 } { a } $, arguably this made it more complicated.
• It's hard to motivate the lemma, which was obtained from OP's AM-GM.