Existence of real roots of a quartic polynomial
Solution 1:
It seems to me that requesting the smallest possible value for $a^2+b^2$ is almost a red herring.
Claim.
For $|a| \leq 4$ and $b > 2 |a| - 2$, the polynomial $$ p(x) = x^4 + ax^3 + bx^2 + ax + 1 $$ has no real zeros.
Proof.
First set $b = 2|a| - 2 + \epsilon$, where $\epsilon > 0$. If we denote $\pm = \operatorname{sign} a$, so that $a = \pm |a|$, we have
$$ \begin{align} p(x) &= x^4 + ax^3 + (2|a|-2+\epsilon)x^2 + ax + 1 \\ &= x^4 - 2x^2 + 1 + x(ax^2 + 2|a|x + a) + \epsilon x^2 \\ &= (x^2-1)^2 + ax(x \pm 1)^2 + \epsilon x^2 \\ &= (x \pm 1)^2\left[ (x \mp 1)^2 + ax \right] + \epsilon x^2 \\ &= (x \pm 1)^2\left[ x^2 \pm (|a|-2)x + 1 \right] + \epsilon x^2 \\ &\geq (x \pm 1)^2 \min\left\{x^2 - 2x + 1,x^2+2x+1\right\} + \epsilon x^2 \\ &= (x \pm 1)^2\min\left\{(x-1)^2,(x+1)^2\right\} + \epsilon x^2 \\ &> 0. \end{align} $$
Q.E.D.
The points on the boundary of the region
$$ A = \{(a,b) \,\colon |a| \leq 4 \,\,\,\text{and}\,\,\,b > 2 |a| - 2\} $$
which have the smallest norm are
$$ (a,b) = \left(\pm \frac{4}{5},-\frac{2}{5}\right), $$
as shown in the following image.
It only remains to demonstrate that one of these points yields a polynomial with a real zero. In fact they both do:
$$ x^4 + \frac{4}{5} x^3 - \frac{2}{5} x^2 + \frac{4}{5} x + 1 $$
has a zero at $x=-1$ and
$$ x^4 - \frac{4}{5} x^3 - \frac{2}{5} x^2 - \frac{4}{5} x + 1 $$
has a zero at $x=1$.
Thus
The smallest value of $a^2 + b^2$ for which we can find some $a,b\in \mathbb R$ such that $p(x)$ has a real zero is $$ \left(\frac{4}{5}\right)^2 + \left(\frac{2}{5}\right)^2 = \frac{4}{5}. $$
Below is a plot of the $(a,b)$-plane which shows the region (in blue) where the polynomial $p(x)$ has at least one real root. Note that for $|a| \leq 4$ the boundary of this region is precisely $b = 2|a| - 2$ (shown in black), but for $|a| > 4$ it curves inward.