Find parameters $a,b$ such that $x^6-2 x^5+2 x^4+2 x^3-x^2-2 x+1-\left(x^3-x^2+a x+b\right)^2>0$

Solution 1:

For the original problem, observe that

$$x^6 - 2x^5 + 2x^4 + 2x^3 - x^2 - 2x + 1 = (x^3 - x^2)^ 2 + (x^2 +x - 1)^2.$$

Since both squares cannot be 0 at the same time, hence it is strictly positive.

For your stated problem, $ a, b = 0 $ is still not a solution to your strict inequality because the expression does hit 0 when $x^2 + x - 1 = 0 $.
Testing values near that point indicates that $ a = -0.01, b = 0.01$ works.
The resulting quartic has 4 complex roots, hence is strictly positive.
Proof by Wolfram Alpha since it's not that interesting.
Given the above, I'm not motivated to figure out the rest of the SOS.

Notes

• You do not always have to "reduce the degree of the polynomial gradually".
  • EG We could have (say) $ f(x) = (0.6x^3 + g(x) ) ^2 + (0.8 x^3 + h(x) ) ^2 $ for some quadratics $g(x), h(x)$.
  • EG Had you tried to do so, you'd likely have ended up with $ (x^3 - x^2 + 0.5 x + 1.5)^2$ in order to remove the $x^4$ and then $x^3 $ term. However, this leaves us with $1.75x^2 - 3.5x - 1.25$ which has distinct real roots, so we can't SOS further.
• I chanced upon this identity, in part from recognizing some of the terms. You guessed that $(x^3-x^2)^2$ might be involved, so looking at $ f(x) - (x^3 -x^2)^2$ would have been a good next step.
• The minimum of $f(x)$ occurs near the positive root of $x^2 + x - 1 = 0$. This shouldn't be too surprising based on the identity.

Solution 2:

Let $p(x)=x^6-2x^5+2x^4+2x^3-x^2-2x+1$. A possible way to show $p(x)>0$ for all $x\in\mathbb{R}$ follows. Consider 3 cases.

Case 1: $x<-1$. Let $r(x)=p(x-1)$. Then $r(x)=x^6-8x^5+27x^4-46x^3+40x^2-18x+5$. Clearly, $r(x)>0$ for $x<0$, so $p(x)=r(x+1)>0$ for $x<-1$.

Case 2: $x>\frac 12$. Let $q(x)=p\left(x+\frac 12\right)$. Then $q(x)=x^6+x^5+\frac34x^4+\frac72x^3+\frac{55}{16}x^2-\frac{15}{16}x+\frac{5}{64}$. We have $\frac{55}{16}x^2-\frac{15}{16}x+\frac{5}{64}>0$ (check by completing the square or computing the discriminant) so $q(x)>0$ for $x>0$ which implies that $p(x)=q\left(x-\frac 12\right)>0$ for $x>\frac 12$.

Case 3: $-1\le x\le \frac 12$. Write $p(x)=x^4(x^2-2x+2)+(2x-1)(x^2-1)$. We have $x^4(x^2-2x+2)=x^4((x-1)^2+1)\ge 0$ with equality only if $x=0$ and $(2x-1)(x^2-1)\ge 0$ for $-1\le x\le \frac 12$ with equality only if $x=\frac 12$ or $x=-1$, so $p(x)>0$ in this case as well.