How to find ${\large\int}_0^1\frac{\ln^3(1+x)\ln x}x\mathrm dx$

Start with integration by parts (IBP) by setting $u=\ln^3(1+x)$ and $dv=\dfrac{\ln x}{x}\ dx$ yields \begin{align} I&=-\frac32\int_0^1\frac{\ln^2(1+x)\ln^2 x}{1+x}\ dx\\ &=-\frac32\int_1^2\frac{\ln^2x\ln^2 (x-1)}{x}\ dx\quad\Rightarrow\quad\color{red}{x\mapsto1+x}\\ &=-\frac32\int_{\large\frac12}^1\left[\frac{\ln^2x\ln^2 (1-x)}{x}-\frac{2\ln^3x\ln(1-x)}{x}+\frac{\ln^4x}{x}\right]\ dx\quad\Rightarrow\quad\color{red}{x\mapsto\frac1x}\\ &=-\frac32\int_{\large\frac12}^1\frac{\ln^2x\ln^2 (1-x)}{x}\ dx+3\int_{\large\frac12}^1\frac{\ln^3x\ln(1-x)}{x}\ dx-\left.\frac3{10}\ln^5x\right|_{\large\frac12}^1\\ &=-\frac32\color{red}{\int_{\large\frac12}^1\frac{\ln^2x\ln^2 (1-x)}{x}\ dx}+3\int_{\large\frac12}^1\frac{\ln^3x\ln(1-x)}{x}\ dx-\frac3{10}\ln^52. \end{align} Applying IBP again to evaluate the red integral by setting $u=\ln^2(1-x)$ and $dv=\dfrac{\ln^2 x}{x}\ dx$ yields \begin{align} \color{red}{\int_{\large\frac12}^1\frac{\ln^2x\ln^2 (1-x)}{x}\ dx}&=\frac13\ln^52+\frac23\color{blue}{\int_{\large\frac12}^1\frac{\ln^3x\ln (1-x)}{1-x}\ dx}. \end{align}

For the simplicity, let $$ \color{blue}{\mathbf{H}_{m}^{(k)}(x)}=\sum_{n=1}^\infty \frac{H_{n}^{(k)}x^n}{n^m}\qquad\Rightarrow\qquad\color{blue}{\mathbf{H}(x)}=\sum_{n=1}^\infty H_{n}x^n, $$ Introduce a generating function for the generalized harmonic numbers for $|x|<1$ $$ \color{blue}{\mathbf{H}^{(k)}(x)}=\sum_{n=1}^\infty H_{n}^{(k)}x^n=\frac{\operatorname{Li}_k(x)}{1-x}\qquad\Rightarrow\qquad\color{blue}{\mathbf{H}(x)}=-\frac{\ln(1-x)}{1-x} $$ and the following identity $$ H_{n+1}^{(k)}-H_{n}^{(k)}=\frac1{(n+1)^k}\qquad\Rightarrow\qquad H_{n+1}-H_{n}=\frac1{n+1} $$

Let us integrating the indefinite form of the blue integral. \begin{align} \color{blue}{\int\frac{\ln^3x\ln (1-x)}{1-x}\ dx}=&-\int\sum_{n=1}^\infty H_nx^n\ln^3x\ dx\\ =&-\sum_{n=1}^\infty H_n\int x^n\ln^3x\ dx\\ =&-\sum_{n=1}^\infty H_n\frac{\partial^3}{\partial n^3}\left[\int x^n\ dx\right]\\ =&-\sum_{n=1}^\infty H_n\frac{\partial^3}{\partial n^3}\left[\frac{x^{n+1}}{n+1}\right]\\ =&-\sum_{n=1}^\infty H_n\left[\frac{x^{n+1}\ln^3x}{n+1}-\frac{3x^{n+1}\ln^2x}{(n+1)^2}+\frac{6x^{n+1}\ln x}{(n+1)^3}-\frac{6x^{n+1}}{(n+1)^4}\right]\\ =&-\ln^3x\sum_{n=1}^\infty \frac{H_{n+1}x^{n+1}}{n+1}+\ln^3x\sum_{n=1}^\infty \frac{x^{n+1}}{(n+1)^2}+3\ln^2x\sum_{n=1}^\infty \frac{H_{n+1}x^{n+1}}{(n+1)^2}\\&-3\ln^2x\sum_{n=1}^\infty \frac{x^{n+1}}{(n+1)^3}-6\ln x\sum_{n=1}^\infty \frac{H_{n+1}x^{n+1}}{(n+1)^3}+6\ln x\sum_{n=1}^\infty \frac{x^{n+1}}{(n+1)^4}\\&+6\sum_{n=1}^\infty \frac{H_{n+1}x^{n+1}}{(n+1)^4}-6\sum_{n=1}^\infty \frac{x^{n+1}}{(n+1)^5}\\ =&\ -\sum_{n=1}^\infty\left[\frac{H_nx^{n}\ln^3x}{n}-\frac{x^{n}\ln^3x}{n^2}-\frac{3H_nx^{n}\ln^2x}{n^2}+\frac{3x^{n}\ln^2x}{n^3}\right.\\& \left.\ +\frac{6H_nx^{n}\ln x}{n^3}-\frac{6x^{n}\ln x}{n^4}-\frac{6H_nx^{n}}{n^4}+\frac{6x^{n}}{n^5}\right]\\ =&\ -\color{blue}{\mathbf{H}_{1}(x)}\ln^3x+\operatorname{Li}_2(x)\ln^3x+3\color{blue}{\mathbf{H}_{2}(x)}\ln^2x-3\operatorname{Li}_3(x)\ln^2x\\&\ -6\color{blue}{\mathbf{H}_{3}(x)}\ln x+6\operatorname{Li}_4(x)\ln x+6\color{blue}{\mathbf{H}_{4}(x)}-6\operatorname{Li}_5(x). \end{align} Therefore \begin{align} \color{blue}{\int_{\Large\frac12}^1\frac{\ln^3x\ln (1-x)}{1-x}\ dx} =&\ 6\color{blue}{\mathbf{H}_{4}(1)}-6\operatorname{Li}_5(1)-\left[\color{blue}{\mathbf{H}_{1}\left(\frac12\right)}\ln^32-\operatorname{Li}_2\left(\frac12\right)\ln^32\right.\\&\left.\ +3\color{blue}{\mathbf{H}_{2}\left(\frac12\right)}\ln^22-3\operatorname{Li}_3\left(\frac12\right)\ln^22+6\color{blue}{\mathbf{H}_{3}\left(\frac12\right)}\ln 2\right.\\&\ -6\operatorname{Li}_4(x)\ln 2+6\color{blue}{\mathbf{H}_{4}(x)}-6\operatorname{Li}_5(x)\bigg]\\ =&\ 12\zeta(5)-\pi^2\zeta(3)+\frac{3}8\zeta(3)\ln^22-\frac{\pi^4}{120}\ln2-\frac{1} {4}\ln^52\\&\ -6\color{blue}{\mathbf{H}_{4}\left(\frac12\right)}+6\operatorname{Li}_4\left(\frac12\right)\ln 2+6\operatorname{Li}_5\left(\frac12\right). \end{align} Using the similar approach as calculating the blue integral, then \begin{align} \int\frac{\ln^3x\ln (1-x)}{x}\ dx&=-\int\sum_{n=1}^\infty \frac{x^{n-1}}{n}\ln^3x\ dx\\ &=-\sum_{n=1}^\infty \frac{1}{n}\int x^{n-1}\ln^3x\ dx\\ &=-\sum_{n=1}^\infty \frac{1}{n}\frac{\partial^3}{\partial n^3}\left[\int x^{n-1}\ dx\right]\\ &=-\sum_{n=1}^\infty \frac{1}{n}\frac{\partial^3}{\partial n^3}\left[\frac{x^{n}}{n}\right]\\ &=-\sum_{n=1}^\infty \frac{1}{n}\left[\frac{x^{n}\ln^3x}{n}-\frac{3x^{n}\ln^2x}{n^2}+\frac{6x^{n}\ln x}{n^3}-\frac{6x^{n}}{n^4}\right]\\ &=\sum_{n=1}^\infty \left[-\frac{x^{n}\ln^3x}{n^2}+\frac{3x^{n}\ln^2x}{n^3}-\frac{6x^{n}\ln x}{n^4}+\frac{6x^{n}}{n^5}\right]\\ &=6\operatorname{Li}_5(x)-6\operatorname{Li}_4(x)\ln x+3\operatorname{Li}_3(x)\ln^2x-\operatorname{Li}_2(x)\ln^3x. \end{align} Hence $$ \int_{\large\frac{1}{2}}^1\frac{\ln^3x\ln (1-x)}{x}\ dx=\frac{\pi^2}{6}\ln^32-\frac{21}{8}\zeta(3)\ln^22-6\operatorname{Li}_4\left(\frac{1}{2}\right)\ln2-6\operatorname{Li}_5\left(\frac{1}{2}\right)+6\zeta(5). $$ Combining altogether, we have

\begin{align} I=&\ \frac{\pi^4}{120}\ln2-\frac{33}4\zeta(3)\ln^22+\frac{\pi^2}2\ln^32-\frac{11}{20}\ln^52+6\zeta(5)+\pi^2\zeta(3)\\ &\ +6\color{blue}{\mathbf{H}_{4}\left(\frac12\right)}-18\operatorname{Li}_4\left(\frac12\right)\ln2-24\operatorname{Li}_5\left(\frac12\right). \end{align}

Continuing my answer in: A sum containing harmonic numbers $\displaystyle\sum_{n=1}^\infty\frac{H_n}{n^3\,2^n}$, we have \begin{align} \color{blue}{\mathbf{H}_{3}\left(x\right)}=&\frac12\zeta(3)\ln x-\frac18\ln^2x\ln^2(1-x)+\frac12\ln x\left[\color{blue}{\mathbf{H}_{2}\left(x\right)}-\operatorname{Li}_3(x)\right]\\&+\operatorname{Li}_4(x)-\frac{\pi^2}{12}\operatorname{Li}_2(x)-\frac12\operatorname{Li}_3(1-x)\ln x+\frac{\pi^4}{60}.\tag1 \end{align} Dividing $(1)$ by $x$ and then integrating yields $$\small\begin{align} \color{blue}{\mathbf{H}_{4}\left(x\right)}=&\frac14\zeta(3)\ln^2 x-\frac18\int\frac{\ln^2x\ln^2(1-x)}x\ dx+\frac12\int\frac{\ln x}x\bigg[\color{blue}{\mathbf{H}_{2}\left(x\right)}-\operatorname{Li}_3(x)\bigg]\ dx\\&+\operatorname{Li}_5(x)-\frac{\pi^2}{12}\operatorname{Li}_3(x)-\frac12\int\frac{\operatorname{Li}_3(1-x)\ln x}x\ dx+\frac{\pi^4}{60}\ln x\\ =&\frac14\zeta(3)\ln^2 x+\frac{\pi^4}{60}\ln x+\operatorname{Li}_5(x)-\frac{\pi^2}{12}\operatorname{Li}_3(x)-\frac18\color{red}{\int\frac{\ln^2x\ln^2(1-x)}x\ dx}\\&+\frac12\left[\color{purple}{\sum_{n=1}^\infty\frac{H_{n}}{n^2}\int x^{n-1}\ln x\ dx}-\color{green}{\int\frac{\operatorname{Li}_3(x)\ln x}x\ dx}-\color{orange}{\int\frac{\operatorname{Li}_3(1-x)\ln x}x\ dx}\right].\tag2 \end{align}$$ Evaluating the red integral using the same technique as the previous one yields \begin{align} \color{red}{\int\frac{\ln^2x\ln^2(1-x)}x\ dx}&=\frac13\ln^3x\ln^2(1-x)-\frac23\color{blue}{\int\frac{\ln(1-x)\ln^3 x}{1-x}\ dx}. \end{align} Evaluating the purple integral yields \begin{align} \color{purple}{\sum_{n=1}^\infty\frac{H_{n}}{n^2}\int x^{n-1}\ln x\ dx}&=\sum_{n=1}^\infty\frac{H_{n}}{n^2}\frac{\partial}{\partial n}\left[\int x^{n-1}\ dx\right]\\ &=\sum_{n=1}^\infty\frac{H_{n}}{n^2}\left[\frac{x^n\ln x}{n}-\frac{x^n}{n^2}\right]\\ &=\color{blue}{\mathbf{H}_{3}(x)}\ln x-\color{blue}{\mathbf{H}_{4}(x)}. \end{align} Evaluating the green integral using IBP by setting $u=\ln x$ and $dv=\dfrac{\operatorname{Li}_3(x)}{x}\ dx$ yields \begin{align} \color{green}{\int\frac{\operatorname{Li}_3(x)\ln x}x\ dx}&=\operatorname{Li}_4(x)\ln x-\int\frac{\operatorname{Li}_4(x)}x\ dx\\ &=\operatorname{Li}_4(x)\ln x-\operatorname{Li}_5(x). \end{align} Evaluating the orange integral using IBP by setting $u=\operatorname{Li}_3(1-x)$ and $dv=\dfrac{\ln x}{x}\ dx$ yields \begin{align} \color{orange}{\int\frac{\operatorname{Li}_3(1-x)\ln x}x\ dx}&=\frac12\operatorname{Li}_3(1-x)\ln^2 x+\frac12\color{maroon}{\int\frac{\operatorname{Li}_2(1-x)\ln^2 x}{1-x}\ dx}. \end{align} Applying IBP again to evaluate the maroon integral by setting $u=\operatorname{Li}_2(1-x)$ and $$ dv=\dfrac{\ln^2 x}{1-x}\ dx\quad\Rightarrow\quad v=2\operatorname{Li}_3(x)-2\operatorname{Li}_2(x)\ln x-\ln(1-x)\ln^2x, $$ we have $$\small{\begin{align} \color{maroon}{\int\frac{\operatorname{Li}_2(1-x)\ln^2 x}{1-x}\ dx}=&\left[2\operatorname{Li}_3(x)-2\operatorname{Li}_2(x)\ln x-\ln(1-x)\ln^2x\right]\operatorname{Li}_2(1-x)\\ &-2\int\frac{\operatorname{Li}_3(x)\ln x}{1-x}\ dx+2\int\frac{\operatorname{Li}_2(x)\ln x}{1-x}\ dx+\color{blue}{\int\frac{\ln(1-x)\ln^3 x}{1-x}\ dx}. \end{align}}$$

We use the generating function for the generalized harmonic numbers evaluate the above integrals involving polylogarithm.

\begin{align} \int\frac{\operatorname{Li}_k(x)\ln x}{1-x}\ dx&=\sum_{n=1}^\infty H_{n}^{(k)}\int x^n\ln x\ dx\\ &=\sum_{n=1}^\infty H_{n}^{(k)}\frac{\partial}{\partial n}\left[\int x^n\ dx\right]\\ &=\sum_{n=1}^\infty H_{n}^{(k)}\left[\frac{x^{n+1}\ln x}{n+1}-\frac{x^{n+1}}{(n+1)^2}\right]\\ &=\sum_{n=1}^\infty\left[\frac{H_{n+1}^{(k)}x^{n+1}\ln x}{n+1}-\frac{x^{n+1}\ln x}{(n+1)^{k+1}}-\frac{H_{n+1}^{(k)}x^{n+1}}{(n+1)^2}+\frac{x^{n+1}}{(n+1)^{k+2}}\right]\\ &=\sum_{n=1}^\infty\left[\frac{H_{n}^{(k)}x^{n}\ln x}{n}-\frac{x^{n}\ln x}{n^{k+1}}-\frac{H_{n}^{(k)}x^{n}}{n^2}+\frac{x^{n}}{n^{k+2}}\right]\\ &=\color{blue}{\mathbf{H}_{1}^{(k)}(x)}\ln x-\operatorname{Li}_{k+1}(x)\ln x-\color{blue}{\mathbf{H}_{2}^{(k)}(x)}+\operatorname{Li}_{k+2}(x). \end{align}

Dividing generating function of $\color{blue}{\mathbf{H}^{(k)}(x)}$ by $x$ and then integrating yields

\begin{align} \sum_{n=1}^\infty \frac{H_{n}^{(k)}x^n}{n}&=\int\frac{\operatorname{Li}_k(x)}{x(1-x)}\ dx\\ \color{blue}{\mathbf{H}_{1}^{(k)}(x)}&=\int\frac{\operatorname{Li}_k(x)}{x}\ dx+\int\frac{\operatorname{Li}_k(x)}{1-x}\ dx\\ &=\operatorname{Li}_{k+1}(x)+\int\frac{\operatorname{Li}_k(x)}{1-x}\ dx. \end{align}

Repeating the process above yields

\begin{align} \sum_{n=1}^\infty \frac{H_{n}^{(k)}x^n}{n^2} &=\int\frac{\operatorname{Li}_{k+1}(x)}{x}\ dx+\int\frac{\operatorname{Li}_k(x)}{x(1-x)}\ dx\\ \color{blue}{\mathbf{H}_{2}^{(k)}(x)}&=\operatorname{Li}_{k+2}(x)+\operatorname{Li}_{k+1}(x)+\int\frac{\operatorname{Li}_k(x)}{1-x}\ dx, \end{align}

where it is easy to show by using IBP that

\begin{align} \int\frac{\operatorname{Li}_2(x)}{1-x}\ dx&=-\int\frac{\operatorname{Li}_2(1-x)}{x}\ dx\\ &=2\operatorname{Li}_3(x)-2\operatorname{Li}_2(x)\ln(x)-\operatorname{Li}_2(1-x)\ln x-\ln (1-x)\ln^2x \end{align}

and

$$ \int\frac{\operatorname{Li}_3(x)}{1-x}\ dx=-\int\frac{\operatorname{Li}_3(1-x)}{x}\ dx=-\frac12\operatorname{Li}_2^2(1-x)-\operatorname{Li}_3(1-x)\ln x. $$

Now, all unknown terms have been obtained. Putting altogether to $(2)$, we have $$\small{\begin{align} \color{blue}{\mathbf{H}_{4}(x)} =&\ \frac1{10}\zeta(3)\ln^2 x+\frac{\pi^4}{150}\ln x-\frac{\pi^2}{30}\operatorname{Li}_3(x)-\frac1{60}\ln^3x\ln^2(1-x)+\frac65\operatorname{Li}_5(x)\\&-\frac15\left[\operatorname{Li}_3(x)-\operatorname{Li}_2(x)\ln x-\frac12\ln(1-x)\ln^2x\right]\operatorname{Li}_2(1-x)-\frac15\operatorname{Li}_4(x)\\&-\frac35\operatorname{Li}_4(x)\ln x+\frac15\operatorname{Li}_3(x)\ln x+\frac15\operatorname{Li}_3(x)\ln^2x-\frac1{10}\operatorname{Li}_3(1-x)\ln^2 x\\&-\frac1{15}\operatorname{Li}_2(x)\ln^3x-\frac15\color{blue}{\mathbf{H}_{2}^{(3)}(x)}+\frac15\color{blue}{\mathbf{H}_{2}^{(2)}(x)} +\frac15\color{blue}{\mathbf{H}_{1}^{(3)}(x)}\ln x\\&-\frac15\color{blue}{\mathbf{H}_{1}^{(2)}(x)}\ln x+\frac25\color{blue}{\mathbf{H}_{3}(x)}\ln x-\frac15\color{blue}{\mathbf{H}_{2}(x)}\ln^2x+\frac1{15}\color{blue}{\mathbf{H}_{1}(x)}\ln^3x+C.\tag3 \end{align}}$$ The next step is finding the constant of integration. Setting $x=1$ to $(3)$ yields $$\small{\begin{align} \color{blue}{\mathbf{H}_{4}(1)} &=-\frac{\pi^2}{30}\operatorname{Li}_3(1)+\frac65\operatorname{Li}_5(1)-\frac15\operatorname{Li}_4(1)-\frac15\color{blue}{\mathbf{H}_{2}^{(3)}(1)}+\frac15\color{blue}{\mathbf{H}_{2}^{(2)}(1)}+C\\ 3\zeta(5)+\zeta(2)\zeta(3)&=-\frac{\pi^2}{30}\operatorname{Li}_3(1)+\frac{19}{30}\operatorname{Li}_5(1)+\frac{3}{5}\operatorname{Li}_3(1)+C\\ C&=\frac{\pi^4}{450}+\frac{\pi^2}{5}\zeta(3)-\frac35\zeta(3)+3\zeta(5). \end{align}}$$ Thus $$\small{\begin{align} \color{blue}{\mathbf{H}_{4}(x)} =&\ \frac1{10}\zeta(3)\ln^2 x+\frac{\pi^4}{150}\ln x-\frac{\pi^2}{30}\operatorname{Li}_3(x)-\frac1{60}\ln^3x\ln^2(1-x)+\frac65\operatorname{Li}_5(x)\\&-\frac15\left[\operatorname{Li}_3(x)-\operatorname{Li}_2(x)\ln x-\frac12\ln(1-x)\ln^2x\right]\operatorname{Li}_2(1-x)-\frac15\operatorname{Li}_4(x)\\&-\frac35\operatorname{Li}_4(x)\ln x+\frac15\operatorname{Li}_3(x)\ln x+\frac15\operatorname{Li}_3(x)\ln^2x-\frac1{10}\operatorname{Li}_3(1-x)\ln^2 x\\&-\frac1{15}\operatorname{Li}_2(x)\ln^3x-\frac15\color{blue}{\mathbf{H}_{2}^{(3)}(x)}+\frac15\color{blue}{\mathbf{H}_{2}^{(2)}(x)} +\frac15\color{blue}{\mathbf{H}_{1}^{(3)}(x)}\ln x\\&-\frac15\color{blue}{\mathbf{H}_{1}^{(2)}(x)}\ln x+\frac25\color{blue}{\mathbf{H}_{3}(x)}\ln x-\frac15\color{blue}{\mathbf{H}_{2}(x)}\ln^2x+\frac1{15}\color{blue}{\mathbf{H}_{1}(x)}\ln^3x\\&+\frac{\pi^4}{450}+\frac{\pi^2}{5}\zeta(3)-\frac35\zeta(3)+3\zeta(5)\tag4 \end{align}}$$ and setting $x=\frac12$ to $(4)$ yields \begin{align} \color{blue}{\mathbf{H}_{4}\left(\frac12\right)}=&\ \frac{\ln^52}{40}-\frac{\pi^2}{36}\ln^32+\frac{\zeta(3)}{2}\ln^22-\frac{\pi^2}{12}\zeta(3)\\&+\frac{\zeta(5)}{32}-\frac{\pi^4}{720}\ln2+\operatorname{Li}_4\left(\frac12\right)\ln2+2\operatorname{Li}_5\left(\frac12\right).\tag5 \end{align}

Finally, we obtain

\begin{align} \int_0^1\frac{\ln^3(1+x)\ln x}x\ dx=&\ \color{blue}{\frac{\pi^2}2\zeta(3)+\frac{99}{16}\zeta(5)-\frac25\ln^52+\frac{\pi^2}3\ln^32-\frac{21}4\zeta(3)\ln^22}\\&\color{blue}{-12\operatorname{Li}_4\left(\frac12\right)\ln2-12\operatorname{Li}_5\left(\frac12\right)}, \end{align}

which again matches @Cleo's answer.

References :

$[1]\ $ Harmonic number

$[2]\ $ Polylogarithm

Indeed, there is a closed form for this integral: $$I=\frac{\pi^2}3\ln^32-\frac25\ln^52+\frac{\pi^2}2\zeta(3)+\frac{99}{16}\zeta(5)-\frac{21}4\zeta(3)\ln^22\\-12\operatorname{Li}_4\left(\frac12\right)\ln2-12\operatorname{Li}_5\left(\frac12\right).$$

This is an updated partial answer that is rather similar to Jack D'Aurizio's approach. (I really hope he doesn't mind.)

Step 1: Expressing the integral as a sum.

It is easy to derive the formula $$\left(\sum^{\infty}_{n=1}a_nx^n\right)\left(\sum^{\infty}_{n=1}b_nx^n\right)=\sum^\infty_{n=1}\sum^{n}_{k=1}a_kb_{n-k+1}x^{n+1}$$ We apply this formula to derive the Taylor series of $\ln^2(1+x)$. \begin{align} \ln^2(1+x) &=\left(\sum^{\infty}_{n=1}\frac{(-1)^{n-1}}{n}x^n\right)\left(\sum^{\infty}_{n=1}\frac{(-1)^{n-1}}{n}x^n\right)\\ &=\sum^\infty_{n=1}\sum^n_{k=1}\frac{(-1)^{k-1}(-1)^{n-k}}{k(n-k+1)}x^{n+1}\\ &=\sum^\infty_{n=1}\frac{(-1)^{n+1}}{n+1}\sum^n_{k=1}\left(\frac{1}{k}+\frac{1}{n-k+1}\right)x^{n+1}\\ &=\sum^\infty_{n=1}\frac{(-1)^{n+1}2H_n}{n+1}x^{n+1} \end{align} Apply this formula again to obtain the Taylor series of $\displaystyle\frac{\ln^2(1+x)}{1+x}$. \begin{align} \frac{\ln^2(1+x)}{1+x} &=\left(\sum^\infty_{n=1}\frac{(-1)^{n+1}2H_n}{n+1}x^{n+1}\right)\left(\sum^{\infty}_{n=1}(-1)^{n-1}x^{n-1}\right)\\ &=\sum^\infty_{n=1}\sum^n_{k=1}\frac{(-1)^{k+1}(-1)^{n-k}2H_k}{k+1}x^{n+1}\\ &=\sum^\infty_{n=1}2(-1)^{n+1}\sum^n_{k=1}\frac{H_k}{k+1}x^{n+1}\\ \end{align} The inner sum is \begin{align} \sum^n_{k=1}\frac{H_k}{k+1} &=\sum^n_{k=1}\frac{H_{k+1}}{k+1}-\sum^n_{k=1}\frac{1}{(k+1)^2}\\ &=\sum^{n+1}_{k=1}\frac{H_k}{k}-H_{n+1}^{(2)}\\ &=\sum^{n+1}_{k=1}\frac{1}{k}\sum^k_{j=1}\frac{1}{j}-H_{n+1}^{(2)}\\ &=\sum^{n+1}_{j=1}\frac{1}{j}\left(\sum^{n+1}_{k=1}\frac{1}{k}-\sum^{j-1}_{k=1}\frac{1}{k}\right)-H_{n+1}^{(2)}\\ &=H_{n+1}^2-\sum^{n+1}_{j=1}\frac{H_j}{j}\\ &=\frac{H_{n+1}^2-H_{n+1}^{(2)}}{2} \end{align} Hence $$\frac{\ln^2(1+x)}{1+x}=\sum^\infty_{n=1}(-1)^{n+1}\left(H_{n+1}^2-H_{n+1}^{(2)}\right)x^{n+1}$$ Pluck this into the integral. \begin{align} \int^1_0\frac{\ln^3(1+x)\ln{x}}{x}{\rm d}x &=-\frac{3}{2}\int^1_0\frac{\ln^2(1+x)\ln^2{x}}{1+x}{\rm d}x\\ &=-\frac{3}{2}\sum^\infty_{n=1}(-1)^{n+1}\left(H_{n+1}^2-H_{n+1}^{(2)}\right)\int^1_0x^{n+1}\ln^2{x} \ {\rm d}x\\ &=-3\sum^\infty_{n=1}\frac{(-1)^{n+1}\left(H_{n+1}^2-H_{n+1}^{(2)}\right)}{(n+2)^3}\\ &=3\sum^\infty_{n=1}\frac{(-1)^{n}\left(H_{n}^{(2)}-H_{n}^2\right)}{(n+1)^3}\\ \end{align}

Step 2: Evaluation of $\displaystyle\sum^\infty_{n=1}\frac{(-1)^nH_n^{(2)}}{(n+1)^3}$

We begin with some simple manipulations of the sum. \begin{align} \sum^\infty_{n=1}\frac{(-1)^nH_n^{(2)}}{(n+1)^3} &=\sum^\infty_{n=1}\frac{(-1)^nH_{n+1}^{(2)}}{(n+1)^3}-\sum^\infty_{n=1}\frac{(-1)^n}{(n+1)^5}\\ &=-\frac{15}{16}\zeta(5)-\underbrace{\sum^\infty_{n=1}\frac{(-1)^nH_n^{(2)}}{n^3}}_{S} \end{align} Consider the function $\displaystyle f(z)=\frac{\pi\csc(\pi z)\psi_1(-z)}{z^3}$. At the positive integers, \begin{align} {\rm Res}(f,n) &=\operatorname*{Res}_{z=n}\left[\frac{(-1)^n}{z^3(z-n)^3}+\frac{(-1)^n(H_n^{(2)}+2\zeta(2))}{z^3(z-n)}\right]\\ &=\frac{6(-1)^n}{n^5}+\frac{(-1)^nH_n^{(2)}}{n^3}+\frac{2(-1)^n\zeta(2)}{n^3} \end{align} Summing them up gives $$\sum^\infty_{n=1} {\rm Res}(f,n)=-\frac{45}{8}\zeta(5)+S-\frac{3}{2}\zeta(2)\zeta(3)$$ At the negative integers, \begin{align} {\rm Res}(f,-n) &=-\frac{(-1)^n\psi_1(n)}{n^3}\\ &=\frac{(-1)^nH_n^{(2)}}{n^3}-\frac{(-1)^n\zeta(2)}{n^3}-\frac{(-1)^n}{n^5} \end{align} Summing them up gives $$\sum^\infty_{n=1} {\rm Res}(f,-n)=S+\frac{3}{4}\zeta(2)\zeta(3)+\frac{15}{16}\zeta(5)$$ At $z=0$, \begin{align} {\rm Res}(f,0) &=[z^2]\left(\frac{1}{z}+\zeta(2)z\right)\left(\frac{1}{z^2}+\zeta(2)+2\zeta(3)z+3\zeta(4)z^2+4\zeta(5)z^3\right)\\ &=4\zeta(5)+2\zeta(2)\zeta(3) \end{align} Since the sum of the reisudes $=0$, $$\sum^\infty_{n=1}\frac{(-1)^nH_n^{(2)}}{(n+1)^3}=-\frac{41}{32}\zeta(5)+\frac{5}{8}\zeta(2)\zeta(3)$$

Step 3: Evaluation of $\displaystyle\sum^\infty_{n=1}\frac{(-1)^nH_n^{2}}{(n+1)^3}$

Formula $(45)$ in this page states that this sum is equal to $$4{\rm Li}_5\left(\frac{1}{2}\right)+4{\rm Li}_4\left(\frac{1}{2}\right)\ln{2}+\frac{2}{15}\ln^5{2}-\frac{107}{32}\zeta(5)+\frac{7}{4}\zeta(3)\ln^2{2}-\frac{2}{3}\zeta(2)\ln^2{2}-\frac{3}{8}\zeta(2)\zeta(3)$$ Using a previously derived result is really unsatisfactory for me. Nevertheless, I have not been able to derive this result, as contour integration fails here due to the power of the denominator being odd (which implies that the sum will vanish when I add the residues at the positive and negative integers up). It seems that Tunk-Fey's brilliant approach would be the most viable method to crack this last sum.

Step 4: Obtaining the final result

Combining our previous results, we get \begin{align} &\ \ \ \ \ \small{\int^1_0\frac{\ln^3(1+x)\ln{x}}{x}{\rm d}x}\\ &=\small{3\sum^\infty_{n=1}\frac{(-1)^n\left(H_{n}^{(2)}-H_n^2\right)}{(n+1)^3}}\\ &=\small{3\left(\frac{33}{16}\zeta(5)+\zeta(2)\zeta(3)-4{\rm Li}_5\left(\frac{1}{2}\right)-4{\rm Li}_4\left(\frac{1}{2}\right)\ln{2}-\frac{2}{15}\ln^5{2}-\frac{7}{4}\zeta(3)\ln^2{2}+\frac{2}{3}\zeta(2)\ln^3{2}\right)}\\ &=\small{\frac{99}{16}\zeta(5)+\frac{\pi^2}{2}\zeta(3)-12{\rm Li}_5\left(\frac{1}{2}\right)-12{\rm Li}_4\left(\frac{1}{2}\right)\ln{2}-\frac{2}{5}\ln^5{2}-\frac{21}{4}\zeta(3)\ln^2{2}+\frac{\pi^2}{3}\ln^3{2}} \end{align}

Just a partial answer for now.

We have: $$ I = -\frac{3}{2}\int_{0}^{1}\frac{\log^2(1+x)\log^2 x}{1+x}\,dx$$ and since: $$\log(1+z)=\sum_{n=1}^{+\infty}\frac{(-1)^{n+1}}{n}z^n$$ it follows that: $$ [z^N]\log^2(1+z)=(-1)^{N+1}\sum_{n=1}^{N-1}\frac{1}{n(N-n)}=(-1)^{N+1}\frac{2H_{N-1}}{N},$$ $$\log^2(1+z)=\sum_{n=1}^{+\infty}\frac{2(-1)^{n+1} H_{n-1}}{n}z^{n}.\tag{1}$$ Let we focus now on: $$J_n = \int_{0}^{1}\frac{x^n\log^2 x}{1+x}\,dx=\frac{\partial^2}{\partial n^2}\int_{0}^{1}\frac{x^n}{1+x}\,dx.$$ We have: $$ J_n = \frac{1}{4}\left(H_{n/2}^{(3)}-H_{(n-1)/2}^{(3)}\right),$$ hence: $$ \color{blue}{I = -\frac{3}{4}\sum_{n=1}^{+\infty}\frac{(-1)^{n+1}H_{n-1}\left(H_{n/2}^{(3)}-H_{(n-1)/2}^{(3)}\right)}{n}}.\tag{2}$$ or, by partial summation: $$ \color{purple}{I=-\frac{3}{4}\sum_{n=1}^{+\infty}H_{n/2}^{(3)}(-1)^n\left(\frac{H_n}{n+1}+\frac{H_{n-1}}{n}\right).}\tag{3}$$ Another identity that follows from the Taylor series of $\log^3(1-z)$ is: $$\color{red}{I=3\sum_{n=1}^{+\infty}\frac{(-1)^{n+1}\left(H_n^2-H_n^{(2)}\right)}{(n+1)^3}.}\tag{4}$$