Which method to use to integrate this function?

The us try to get rid of the square root step-by-step: $$\begin{eqnarray*} \int_{0}^{1}(1-x)\sqrt{x}\sqrt{4-x}\,dx&\stackrel{x\mapsto z^2}{=}& 2\int_{0}^{1}z^2(1-z^2)\sqrt{4-z^2}\,dz\\&\stackrel{z\mapsto 2u}{=}&32\int_{0}^{1/2}u^2(1-4u^2)\sqrt{1-u^2}\,du\\&\stackrel{u\mapsto\sin\theta}{=}&32\int_{0}^{\pi/6}\sin^2\theta\cos^2\theta(1-4\sin^2\theta)\,d\theta\\&=&4\int_{0}^{\pi/6}\left[-1+\cos(2\theta)+\cos(4\theta)-\cos(6\theta)\right]\,d\theta.\end{eqnarray*}$$ I guess you may take it from here.

It is always didactically good to give a solution for the given start. I will try it, although trigonometric substitutions from the beginning may save typing.

The substitution $t=\sqrt{(4-x)/x}$ is ok, but we have to use improper integrals. The solution would be as follows. For $x=0+$ the corresponding (limiting) value of $t$ is $t=\sqrt{4/0_+}=+\infty$, and for $x=1$ we get $t=\sqrt{(4-1)/1}=\color{red}{\sqrt 3}$. (It is square three, maybe the one reason for writing this answer.) $$ \begin{aligned} J &= \int_0^1 (1-x)\sqrt{4x-x^2} \; dx \\ &= \int_{\infty}^{\sqrt 3} -32\frac{(t^2-3)t^2}{(1+t^2)^4}\;dt \\ &= +32\int_{\sqrt 3}^{\infty} \frac{(t^2+2t^2+1)-5(t^2+1)+4}{(1+t^2)^4}\;dt \\ &= +32\int_{\sqrt 3}^{\infty} \left[\ \frac1{(1+t^2)^2} -5\frac1{(1+t^2)^3} +4\frac1{(1+t^2)^4} \ \right]\;dt \\ &=32(K_2-5K_3+4K_4)\ , \end{aligned} $$ where $K_n$ is the integral on the same interval of $(1+t^2)^{-n}$. We have the recursion $$ \begin{aligned} K_n &=\int_{\sqrt 3}^{\infty}t'\frac1{(1+t^2)^n}\;dt \\ &=\frac t{(1+t^2)^n}\Bigg|_{\sqrt 3}^{\infty} - \int_{\sqrt 3}^{\infty} t\cdot \frac{-2nt}{(1+t^2)^{n+1}}\;dt \\ &=-\frac{\sqrt3}{4^n} + 2n\int_{\sqrt 3}^{\infty}\frac{(t^2+1)-1}{(1+t^2)^{n+1}}\;dt \\ &=-\frac{\sqrt3}{4^n} + 2n(K_n-K_{n+1})\ ,\text{ i.e.} \\ K_{n+1} &= \frac 1{2n}\left[\ (2n-1)K_n-\frac{\sqrt3}{4^n}\ \right]\ . \\[2mm] &\qquad\text{This gives:} \\ K_1 &=\arctan \infty-\arctan\sqrt 3 =\left(\frac 12-\frac 13\right)\pi =\frac 16\pi\ , \\ K_2 &=\frac 12\left[K_1-\frac{\sqrt 3}4\right]=\frac 1{12}\pi - \frac{\sqrt 3}8\ , \\ K_3 &=\frac 14\left[3K_2-\frac{\sqrt 3}{16}\right] =\frac 1{16}\pi - \frac{7\sqrt 3}{64}\ , \\ K_4 &=\frac 16\left[5K_3-\frac{\sqrt 3}{64}\right] =\frac 5{96}\pi - \frac{3\sqrt 3}{32}\ , %\\ %K_5 &=\frac 18\left[7K_4-\frac{\sqrt 3}{216}\right] %=\frac {35}{768}\pi - \frac{169\sqrt 3}{2048}\ , \\ &\qquad\text{ so putting all together} \\ J&=32(K_2-5K_3+4K_4) \\ &=-\frac23\pi+\frac {3\sqrt 3}2\ . \end{aligned} $$ As said, not the quick way, but all details are displayed. Computer check:

sage: var('x,t');
sage: integral( (1-x)*sqrt(4*x-x^2), x, 0, 1 ).simplify()
-2/3*pi + 3/2*sqrt(3)
sage: integral( 32*(t^2-3)*t^2/(1+t^2)^4, t, sqrt(3), oo ).simplify()
-2/3*pi + 3/2*sqrt(3)

One can use $\,\displaystyle{\int_a^b f(x) dx=\int_a^b f(a+b-x) dx}\,$ to get: $$I=\int_0^1 x \sqrt{4-(1+x)^2} dx= \int_0^1 (x+1)\sqrt{4 - (x+1)^2} dx-\int_0^1 \sqrt{4 - (x+1)^2} dx$$ For the first one just substitute $4-(x+1)^2 =t$ to get $\displaystyle{\frac12 \int_0^3 \sqrt t dt}.\, $ And the second one is a standard square root integral.