How did Euler realize $x^4-4x^3+2x^2+4x+4=(x^2-(2+\alpha)x+1+\sqrt{7}+\alpha)(x^2-(2-\alpha)x+1+\sqrt{7}-\alpha)$?

Euler lived a century after Isaac Newton and Blaise Pascal, so he must have been familiar with the former's binomial theorem and the latter's triangle. Indeed, the polynomial you presented looks quite similar to the binomial expansion of $(x-1)^4$, whose coefficients are found on the fourth row of Pascal's triangle. By subtracting the two, we are left with $4x^2-8x-3$, whose roots are $1\pm\dfrac{\sqrt7}2$ which is a quarter of $\alpha$. So, $$P(x)=(x-1)^4-4\bigg[(x-1)-\dfrac{\sqrt7}2\bigg]\bigg[(x-1)+\dfrac{\sqrt7}2\bigg],$$ which, after substituting $u=(x-1)^2$, becomes $u^2-4u+7$. Then, by completing the square, we arrive at the desired result.

I like this rather old question. Here is a yet another possible way Euler could have taken:

Note that $\displaystyle x^4+ax^2+b$ can be factorized easily if $\displaystyle a^2-4b\geq 0$. If, however, $\displaystyle a^2-4b\leq 0$, then \begin{align} x^4+ax^2+b&=(x^2+\sqrt{b})^2-(x\sqrt{2\sqrt{b}-a})^2\\ &=(x^2+\sqrt{b}-x\sqrt{2\sqrt{b}-a})(x^2+\sqrt{b}+x\sqrt{2\sqrt{b}-a}). \end{align}

Now in $P(x)=x^4-4x^3+2x^2+4x+4$, use $x=y+1$, and proceed: \begin{align} P(y+1)&=y^4-4y^2+7\\ &=(y^2+\sqrt{7}-y\sqrt{2\sqrt{7}+4})(y^2+\sqrt{7}+y\sqrt{2\sqrt{7}+4}). \end{align}

substituting $y=x-1$ we arrive at Euler's result.

The other answers are great but were pure speculation since Euler published how he solves quartics.

From Elements of Algebra by Euler section 4 chapter 15, his new method for resolving fourth order equations. Additionally, Google books has a full copy to download or read online here.

Suppose the root of the equation is of the form $x = \sqrt{p} + \sqrt{q} + \sqrt{r}$ where $p,q,r$ are the roots of an equation of degree three, $$ z^3 -fz^2+gz-h=0 $$ and \begin{align} f&=p+q+r\tag{1}\\ g&=pq+pr+qr\tag{2}\\ h&=pqr\tag{3} \end{align} Now, he square $x$ and obtained $$ x^2 = p+q+r+2\sqrt{pq}+2\sqrt{pr}+2\sqrt{qr} $$ From $(1)$, we can now write $$ x^2-f=2\sqrt{pq}+2\sqrt{pr}+2\sqrt{qr}\tag{4} $$ Let's square $(4)$ again to obtain: $$ x^4-2fx^2+f^2 = 4pq+4pr+4qr+8\sqrt{p^2qr}+8\sqrt{pq^2r}+8\sqrt{pqr^2} $$ From $(2)$, we can now write $$ x^4-2fx^2+f^2 -4g= 8\sqrt{pqr}(\sqrt{p}+\sqrt{q}+\sqrt{r}) $$ Using the identity first laid out for $x$ and $(3)$, we have $$ x^4-2fx^2 -8x\sqrt{h}+f^2-4g= 0\tag{5} $$ Euler says we don't need to worry about $yx^3$ because

for we shall afterwards shew, that every complete equation may be transformed into another, from which the second term is taken away.

Next, Euler goes onto factoring a quartic as a product of two real quadratics.

It is to be observed that the product of these three terms, or $\sqrt{pqr}$, must be equal to $\sqrt{h}=b/8$, and that if $b/8$ is positive, the product of the terms $\sqrt{p},\sqrt{q},\sqrt{r}$; must likewise be positive so that all variations that can be admitted are reduced to the four following:

\begin{align} x &= \sqrt{p} + \sqrt{q} + \sqrt{r}\\ x &= \sqrt{p} - \sqrt{q} - \sqrt{r}\\ x &= -\sqrt{p} + \sqrt{q} - \sqrt{r}\\ x &= -\sqrt{p} - \sqrt{q} + \sqrt{r} \end{align} When $b/8$ is negative, we have \begin{align} x &= \sqrt{p} + \sqrt{q} - \sqrt{r}\\ x &= \sqrt{p} - \sqrt{q} + \sqrt{r}\\ x &= -\sqrt{p} + \sqrt{q} + \sqrt{r}\\ x &= -\sqrt{p} - \sqrt{q} - \sqrt{r} \end{align}

In order to use Euler's method, you would first need to transform your equation such that the $x^3$ term is gone. Then you match up the coefficients of $f,g,h$ in your new polynomial with the general form $(5)$. That is, given a polynomial of the form $$ x^4 - ax^2 - bx - c = 0 $$ you would set $a = 2f$, $b = 8\sqrt{h}$, and $-c = f^2 - 4g$. Then you can follow his book's examples which you will find in the second hyperlink.

Once it is noticed that $$ x^4-4x^3+2x^2+4x+4=(x-1)^4-4(x-1)^2+7 $$ the square can be completed to get $$ \left((x-1)^2-2\right)^2+3=\color{#00A000}{\left((x-1)^2-2-i\sqrt3\right)}\color{#0000FF}{\left((x-1)^2-2+i\sqrt3\right)} $$ Solving for $(\alpha+i\beta)^2=2+i\sqrt3$, we get $\alpha^2-\beta^2=2$ and $2\alpha\beta=\sqrt3$. Adding the squares and taking the square root gives $\alpha^2+\beta^2=\sqrt7$. Solving for $\alpha$ and $\beta$ yields $$ \color{#00A000}{\left({\small\sqrt{\frac{2+\sqrt7}2}}+i\,{\small\sqrt{\frac{-2+\sqrt7}2}}\right)^2=2+i\sqrt3} $$ $$ \color{#0000FF}{\left({\small\sqrt{\frac{2+\sqrt7}2}}-i\,{\small\sqrt{\frac{-2+\sqrt7}2}}\right)^2=2-i\sqrt3} $$ The full factorization is $$ \overset{\underbrace{\color{#0000FF}{\left[x-1-\sqrt{\frac{2+\sqrt7}2}+i\,\sqrt{\frac{-2+\sqrt7}2}\right]}\color{#00A000}{\left[x-1-\sqrt{\frac{2+\sqrt7}2}-i\,\sqrt{\frac{-2+\sqrt7}2}\right]}}_{}} {\left[\left(x-1-\sqrt{\frac{2+\sqrt7}2}\right)^2+\frac{-2+\sqrt7}2\right]} \overset{\underbrace{\color{#00A000}{\left[x-1+\sqrt{\frac{2+\sqrt7}2}+i\,\sqrt{\frac{-2+\sqrt7}2}\right]}\color{#0000FF}{\left[x-1+\sqrt{\frac{2+\sqrt7}2}-i\,\sqrt{\frac{-2+\sqrt7}2}\right]}}_{}} {\left[\left(x-1+\sqrt{\frac{2+\sqrt7}2}\right)^2+\frac{-2+\sqrt7}2\right]} $$ This gives the factorization sought.

Hint:As @illysial, @Mehdi and others indicated, the substitution $x = t+1$ transforms the expression $x^4 − 4x^3 + 2x^2 + 4x + 4$ into $t^4-4 t^2+7$.