$\int_0^1\frac{\ln x\ln^2(1-x^2)}{\sqrt{1-x^2}}dx=\frac{\pi}{2}\zeta(3)-2\pi\ln^32$

As shown in this answer $$ \frac{\Gamma'(x)}{\Gamma(x)}=H_{x-1}-\gamma\tag1 $$ Using that $H_x^{(n)}=\sum\limits_{k=1}^\infty\!\left(\frac1{k^n}-\frac1{(k+x)^n}\right)$, we get $$ \frac{\mathrm{d}}{\mathrm{d}x}H_x^{(n)}=n\!\left(\zeta(n+1)-H_x^{(n+1)}\right)\tag2 $$ In particular, $$ \begin{array}{c|c|c} x&H_x&H_x^{(2)}&H_x^{(3)}\\\hline -\frac12&-2\log(2)&-2\zeta(2)&-6\zeta(3)\\ 0&0&0&0 \end{array} $$ Applying $(1)$ and $(2)$, $$ \begin{align} \Gamma'(x) &=\Gamma(x)(H_{x-1}-\gamma)\tag{3a}\\ \Gamma''(x) &=\Gamma(x)\!\left((H_{x-1}-\gamma)^2+\zeta(2)-H_{x-1}^{(2)}\right)\tag{3b}\\ \Gamma'''(x) &=\Gamma(x)\!\left((H_{x-1}-\gamma)^3+3(H_{x-1}-\gamma)\left(\zeta(2)-H_{x-1}^{(2)}\right)+2\!\left(H_{x-1}^{(3)}-\zeta(3)\right)\right)\tag{3c}\\ \end{align} $$ In particular, $$ \begin{array}{c|c|c} x&\Gamma(x)&\Gamma'(x)&\Gamma''(x)&\Gamma'''(x)\\\hline \frac12&\sqrt\pi&-\sqrt\pi(2\log(2)+\gamma)&\sqrt\pi\!\left((2\log(2)+\gamma)^2+3\zeta(2)\right)&\text{not needed}\\ 1&1&-\gamma&\gamma^2+\zeta(2)&-\gamma^3-3\gamma\zeta(2)-2\zeta(3) \end{array} $$ Now we get to the integral: $$ \begin{align} &\int_0^1\frac{\log(x)\log^2\left(1-x^2\right)}{\sqrt{1-x^2}}\,\mathrm{d}x\\ &=\frac14\int_0^1\frac{\log(x)\log^2\left(1-x\right)}{\sqrt{x(1-x)}}\,\mathrm{d}x\tag{4a}\\[6pt] &=\left.\frac14\frac{\partial}{\partial\alpha}\frac{\partial^2}{\partial\beta^2}\operatorname{B}(\alpha,\beta)\,\right|_{(\alpha,\beta)=\left(\frac12,\frac12\right)}\tag{4b}\\ &=\left.\frac14\frac{\partial}{\partial\alpha}\frac{\partial^2}{\partial\beta^2}\frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}\,\right|_{(\alpha,\beta)=\left(\frac12,\frac12\right)}\tag{4c}\\ &=\scriptsize\left(-6\Gamma(\alpha)\Gamma(\beta) \Gamma'(\alpha+\beta)^3+\Gamma(\alpha+\beta)^3\Gamma'(\alpha)\Gamma''(\beta)\right.\\ &\scriptsize\left.{}+2\Gamma(\alpha+\beta)\Gamma'(\alpha+\beta)\left(2\Gamma(\alpha)\Gamma'(\beta)\Gamma'(\alpha+\beta)+\Gamma(\beta)\left(\Gamma'(\alpha)\Gamma'(\alpha+\beta)+3\Gamma(\alpha)\Gamma''(\alpha+\beta)\right)\right)\right.\\ &\scriptsize\left.{}-\Gamma(\alpha+\beta)^2\left(\Gamma'(\alpha)\left(2\Gamma'(\beta)\Gamma'(\alpha+\beta)+\Gamma(\beta)\Gamma''(\alpha+\beta)\right)\right.\right.\\ &\scriptsize\left.\left.\left.{}+\Gamma(\alpha)\left(\Gamma'(\alpha+\beta)\Gamma''(\beta)+2\Gamma'(\beta)\Gamma''(\alpha+\beta)+\Gamma(\beta)\Gamma'''(\alpha+\beta)\right)\right)\right)\frac1{4\Gamma(\alpha+\beta)^4}\right|_{(\alpha,\beta)=\left(\frac12,\frac12\right)}\tag{4d}\\ &=\left(-6\Gamma\!\left(\tfrac12\right)\Gamma\!\left(\tfrac12\right)\Gamma'(1)^3+\Gamma(1)^3\Gamma'\!\left(\tfrac12\right)\Gamma''\!\left(\tfrac12\right)\right.\\ &\left.{}+2\Gamma(1)\Gamma'(1)\left(2\Gamma\!\left(\tfrac12\right)\Gamma'\!\left(\tfrac12\right)\Gamma'(1)+\Gamma\!\left(\tfrac12\right)\left(\Gamma'\!\left(\tfrac12\right)\Gamma'(1)+3\Gamma\!\left(\tfrac12\right)\Gamma''(1)\right)\right)\right.\\ &\left.{}-\Gamma(1)^2\!\left(\Gamma'\!\left(\tfrac12\right)\left(2\Gamma'\!\left(\tfrac12\right)\Gamma'(1)+\Gamma\!\left(\tfrac12\right)\Gamma''(1)\right)\right.\right.\\ &\left.\left.{}+\Gamma\!\left(\tfrac12\right)\left(\Gamma'(1)\Gamma''\!\left(\tfrac12\right)+2\Gamma'\!\left(\tfrac12\right)\Gamma''(1)+\Gamma\!\left(\tfrac12\right)\Gamma'''(1)\right)\right)\right)\frac1{4\Gamma(1)^4}\tag{4e}\\ &=\frac14\left(6\pi\gamma^3-\pi(2\log(2)+\gamma)^3-\frac{\pi^3}2(2\log(2)+\gamma)\right.\\ &\left.{}-2\gamma\left(6\pi\gamma\log(2)+6\pi\gamma^2+\frac{\pi^3}2\right)\right.\\ &\left.{}-\left(-\pi\left(8\gamma\log^2(2)+10\gamma^2\log(2)+3\gamma^3+(2\log(2)+\gamma)\frac{\pi^2}6\right)\right.\right.\\ &\left.\left.{}-\pi\left(4\gamma(\log(2)+\gamma)^2+(2\log(2)+4\gamma)\frac{\pi^2}3+2\zeta(3)\right)\vphantom{\frac{\pi^2}6}\right)\right)\tag{4f}\\ &=\frac14\left(0\pi\gamma^3+0\pi\gamma^2\log(2)+0\pi\gamma\log^2(2)+0\pi^3\gamma+0\pi^3\log(2)\right)\\ &+\frac14\left(-8\pi\log^3(2)+2\pi\zeta(3)\right)\tag{4g}\\[6pt] &=\bbox[5px,border:2px solid #C0A000]{\frac\pi2\zeta(3)-2\pi\log^3(2)}\tag{4h} \end{align} $$ Explanation:
$\text{(4a)}$: substitute $x\mapsto\sqrt{x}$
$\text{(4b)}$: write as a derivative of the Beta Function
$\text{(4c)}$: write the Beta Function in terms of the Gamma Function
$\text{(4d)}$: take the derivatives (nothing special about $\Gamma$)
$\text{(4e)}$: substitute $(\alpha,\beta)\mapsto\left(\frac12,\frac12\right)$
$\text{(4f)}$: apply the values from the $\Gamma$ table
$\text{(4g)}$: collect like terms
$\text{(4h)}$: simplify

Substitute $t=\sin x$

\begin{align} & \int_0^1\frac{\ln x\ln^2(1-x^2)}{\sqrt{1-x^2}}dx\\ =& \> 4\int_0^{\pi/2} \ln (\sin t)\ln^2(\cos t)dt\\ =& \> 2\int_0^{\pi/2} [\ln (\sin t)\ln^2(\cos t)+ \ln^2 (\sin t)\ln(\cos t)]dt\\ =& \> \frac23\int_0^{\pi/2} [\ln^3(\sin t\cos t)-2 \ln^3 (\sin t)]dt\\ =&\>\frac23 \int_0^{\pi/2} \ln^3\frac{\sin t}2 dt- \frac43 \int_0^{\pi/2} \ln^3 (\sin t)\>dt \\ =&\> - \frac23 \int_0^{\pi/2} \ln^3 (2\sin t)\>dt +4\ln^2 2\int_0^{\pi/2} \ln(2\sin t)dt - 2\pi \ln^32\\ =&\>\frac\pi2\zeta(3) -2\pi\ln^32 \end{align} where $\int_0^{\pi/2} \ln(2\sin t)dt=0$ and $\int_0^{\pi/2} \ln^3 (2\sin t)dt = -\frac{3\pi}4\zeta(3)$.

$$I=\int_0^1\frac{\ln x\ln^2(1-x^2)}{\sqrt{1-x^2}}dx$$ $$=\frac{1}{4}\int_0^1\frac{\ln t\ln^2(1-t)}{\sqrt t\sqrt{1-t}}dt=\frac{1}{4}\frac{d^2}{db^2}\frac{d}{da}\int_0^1t^{a-1/2}(1-t)^{b-1/2}dt|_{a=b=0}$$ $$=\frac{1}{4}\frac{d^2}{db^2}\frac{d}{da}B(a+1/2;b+1/2)|_{a=b=0}$$ $$\frac{1}{4}B(a+1/2;b+1/2)=\frac{1}{4}\frac{\Gamma(a+1/2)\Gamma(b+1/2)}{\Gamma(a+b+1)}=\frac{\pi}{4}\frac{\Gamma(1+2a)}{\Gamma(1+a)}\frac{\Gamma(1+2b)}{\Gamma(1+b)}\frac{4^{-a}4^{-b}}{\Gamma(1+a+b)}$$ $$\Gamma(a)\Gamma\bigl(a+1/2\bigr)=2\sqrt{\pi}\,4^{-a}\,\Gamma(2a);\,\,a\Gamma(a)=\Gamma(1+a)=1-\gamma a+\Gamma''(1)\frac{a^2}{2}+...$$ $$\Psi(t)=\frac{\Gamma'(t)}{\Gamma(t)};\,\,\Psi(1)=-\gamma;\,\,\Gamma''(1)=\gamma^2+\frac{\pi^2}{6};\,\,\Psi'(1)=\zeta(2);\,\,\Psi''(1)=-2\zeta(3)$$ $$I=\frac{\pi}{4}\frac{d^2}{db^2}\frac{d}{da}\biggl(\frac{(1-a\gamma+...)(1-a\ln4+...)}{\Gamma(1+b)+\Gamma'(1+b)a+...}\frac{\Gamma(1+2b)}{\Gamma(1+b)}4^{-b}\biggr)|_{a=b=0}$$ $$=-\frac{\pi}{4}\frac{d^2}{db^2}\biggl(\bigl(1-b\ln4+\frac{b^2}{2}\ln^24-..\bigr)\frac{1-2\gamma b+2\Gamma''(1)b^2+..}{(1-\gamma b+\Gamma''(1)\frac{b^2}{2}+..)^2}\bigl(\gamma+\ln4+\Psi(1+b)\bigr)\biggr)|_{b=0}$$ $$=-\frac{\pi}{4}\frac{d^2}{db^2}\biggl(\bigl(1-b\ln4+\frac{b^2}{2}\ln^24+..\bigr)\bigl(1-2\gamma b+2\Gamma''(1)b^2+..\bigr)\bigl(1+2\gamma b-\Gamma''(1)b^2+3\gamma^2b^2+..\bigr)\bigl(\gamma+\ln4-\gamma+b\zeta(2)-b^2\zeta(3)\bigr)+...\Bigr)|_{b=0}$$ $$=-\frac{\pi}{2}\Bigl(\frac{\pi^2}{6}\ln4-\ln4\zeta(2)-\zeta(3)+\frac{1}{2}\ln^34\Bigr)$$ $$I=\frac{\pi}{2}\zeta(3)-2\pi\ln^32$$