How to calculate $ \int_{0}^{\infty} \frac{ x^2 \log(x) }{1 + x^4} $?

It's a bit more tricky that what you describe, but the general idea is correct. Instead of integrating from $0$ to $\infty$, one can integrate from $-\infty$ to $+\infty$ slightly above the real axis. Because of the logarithm, the integral from $-\infty$ to $0$ will give a possibly non-zero imaginary part, but the real part will be an even function of $x$. So we can write: \begin{align} \int_0^{\infty}\frac{x^2\ln x}{1+x^4}dx&=\frac12\mathrm{Re}\,\int_{-\infty+i0}^{\infty+i0} \frac{x^2\ln x}{1+x^4}dx=\\&=\pi\cdot \mathrm{Re}\left[ i\left(\mathrm{res}_{x=e^{i\pi/4}}\frac{x^2\ln x}{1+x^4}+\mathrm{res}_{x=e^{3i\pi/4}}\frac{x^2\ln x}{1+x^4}\right)\right]=\\ &=\pi\cdot \mathrm{Re}\left[ i\left(\frac{\pi e^{i\pi/4}}{16}- \frac{3\pi e^{3i\pi/4}}{16}\right)\right]=\\ &=\pi\cdot\mathrm{Re}\frac{(1+2i)\pi}{8\sqrt{2}}=\frac{\pi^2}{8\sqrt{2}}. \end{align}

Now as far as I understand the question was about how can one justify the vanishing of the integral over the half-circle $C$ which in its turn justifies the application of residue theorem. Parameterizing that circle as $x=Re^{i\varphi}$, $\varphi\in(0,\pi)$, we see that \begin{align} \int_C \frac{x^2\ln x}{1+x^4}dx=\int_0^{\pi}\frac{iR^3e^{3i\varphi}\left(i\varphi+\ln R\right)}{1+R^4e^{4i\varphi}}d\varphi=O\left(\frac{\ln R}{R}\right), \end{align} which indeed vanishes as $R\rightarrow \infty$.

Recalling the Mellin transform of a function $f$

$$ F(s)=\int_{0}^{\infty}x^{s-1}f(x)dx \implies F'(s) = \int_{0}^{\infty} x^{s-1}\ln(x)f(x) dx.$$

So, taking $f(x)=\frac{1}{x^4+1}$ and finding its Mellin transform

$$ F(s)=\frac{1}{4}\,{\frac {\pi }{\sin \left( \frac{\pi \,s}{4} \right) }} \implies F'(s)=\frac{1}{16}\,{\frac {{\pi }^{2}\cos \left( \pi \,s/4 \right) }{ \left( \cos \left( \pi \,s/4 \right) \right) ^{2}-1}}$$

Taking the limit as $s\to 3$ yields the desired result

$$ \lim_{s\to 3}F'(s)=\frac{\pi^2}{8\sqrt{2}}. $$

$$\underbrace{\dfrac{x^2 \log(x)}{1+x^4} dx \to \dfrac1{x^2} \dfrac{\log(1/x)}{1+1/x^4} \dfrac{-dx}{x^2}}_{x \to 1/x}$$ Hence, \begin{align} I = \int_0^{\infty} \dfrac{x^2 \log(x)}{1+x^4} dx & = \int_0^1 \dfrac{x^2 \log(x)}{1+x^4} dx + \int_1^{\infty} \dfrac{x^2\log(x)}{1+x^4}dx\\ & = \int_0^1 \dfrac{x^2 \log(x)}{1+x^4} dx - \int_0^1 \dfrac{\log(x)}{1+x^4}dx\\ & = \int_0^1 \dfrac{x^2-1}{1+x^4} \log(x) dx \end{align} \begin{align} \int_0^1 \dfrac{x^2 \log(x)}{1+x^4} dx & = \int_0^1 \sum_{k=0}^{\infty}(-1)^k x^{4k+2} \log(x)dx\\ & = \sum_{k=0}^{\infty}(-1)^k \int_{0}^{\infty} x^{4k+2} \log(x)dx\\ & = \sum_{k=0}^{\infty} \dfrac{(-1)^{k+1}}{(4k+3)^2} \end{align} \begin{align} \int_0^1 \dfrac{\log(x)}{1+x^4} dx & = \int_0^1 \sum_{k=0}^{\infty}(-1)^k x^{4k} \log(x)dx\\ & = \sum_{k=0}^{\infty}(-1)^k \int_{0}^{\infty} x^{4k} \log(x)dx\\ & = \sum_{k=0}^{\infty} \dfrac{(-1)^{k+1}}{(4k+1)^2} \end{align} $$I = \sum_{k=0}^{\infty} (-1)^k \left(\dfrac1{(4k+1)^2} - \dfrac1{(4k+3)^2}\right) = \dfrac{\pi^2}{8\sqrt2}$$ which can be obtained from the identity that the PolyLogarithmic function satisfies $$\text{Li}_n(e^{2 \pi ix}) + (-1)^n \text{Li}_n(e^{-2 \pi ix}) = - \dfrac{(2\pi i)^n}{n!}B_n(x)$$ You can see the posts here and here for more details.

Another way to handle these kinds of integrals is by setting $$ f(z) = \frac{z^2 \log^2 z}{1+z^4} $$ where $\log$ is the "natural branch" of the complex logarithm and integrate $f$ along a keyhole contour, see for example this. The idea with this approach is to take advantage of the branch cut of $\log$.

Use the substitution $x^4=t \Rightarrow dx=dt/(4t^{3/4})$ to obtain the following definite integral: $$\frac{1}{16}\int_0^{\infty} \frac{t^{-1/4}\ln t}{1+t}\,dt$$ Consider the result I proved here: Evaluate $\int_0^\infty\!\!\int_0^\infty\!\!\int_0^\infty\!\frac{(xyz)^{-1/7}(yz)^{-1/7}z^{-1/7}}{(x+1)(y+1)(z+1)}dx\,dy\,dz$ which states: $$\int_0^{\infty} \frac{t^{-a}}{1+t}\,dt=\frac{\pi}{\sin (a\pi)}$$ Differentiate the above with respect to $a$ to get: $$\int_0^{\infty} \frac{t^{-a}\ln t}{1+t}dt=\pi^2\cot(a\pi)\csc(a\pi)$$ Substitute $a=1/4$ and divide both the sides by 16 to get: $$\frac{1}{16}\int_0^{\infty} \frac{t^{-1/4}\ln t}{1+t}dt=\frac{\pi^2}{8\sqrt{2}}$$