Why does this $u$-substitution zero out my integral?

You certainly have the right to make any change of variable you want. The problem with using $u=x(x-2)$ is that you have to solve for $x$ as a function of $u$:

$$u=x^2-2 x \implies x = 1 \pm \sqrt{1+u} \implies dx = \pm \frac{du}{2 \sqrt{1+u}} $$

Because $u(x)$ is a quadratic, $x(u)$ is multivalued with two branches. (This is why you were able to get zero in the lower and upper limits.) You would sub differently along each branch. Thus,

$$\int_0^2 dx \frac{x}{\sqrt{1+x^2}} = \frac12 \int_0^{-1} du \frac{1-(1+u)^{-1/2}}{\sqrt{3+u-2 \sqrt{1+u}}} + \frac12 \int_{-1}^0 \frac{1+(1+u)^{-1/2}}{\sqrt{3+u+2 \sqrt{1+u}}}$$

..and take it from there.

The wrong output is due to the illegitimate choice of change of variable function $\phi(x)=x(x-2)$. There are different formulations of the substitution formula for integration, but usually injectivity of $\phi$ is a requirement.

Challenge Change the lower bound of integration from $0$ to $-1$. Now the function $\phi(x)=x^2+1$ is no longer injective over this interval. The challenge question is whether you can still use the substitution $u=x^2+1$ to find the right answer.