A solution by Cornel Ioan Valean

First, let's observe that $$\log ^2\left(\frac{1}{2}\sin(2x)\right)+\log ^2(\tan(x))=2 \log ^2(\sin(x))+2 \log ^2(\cos(x)), \ 0<x<\pi/2, \tag1$$ and if we multiply both side of $(1)$ by $x$ and integrate from $x=0$ to $x=\pi/4$, we get $$\int_0^{\pi/4}x\log ^2\left(\frac{1}{2} \sin(2x)\right)\textrm{d}x+\int_0^{\pi/4}x\log ^2(\tan(x))\textrm{d}x$$ $$=2 \int_0^{\pi/4}x\log ^2(\sin(x))\textrm{d}x+2\int_0^{\pi/4}x \log ^2(\cos(x)) \textrm{d}x. \tag2$$

Since $$\int_0^{\pi/4}x \log ^2\left(\frac{1}{2} \sin(2x)\right)\textrm{d}x=\frac{1}{4}\int_0^{\pi/2}x\log^2\left(\frac{1}{2} \sin(x)\right)\textrm{d}x$$ $$=\frac{\log^2(2)}{4}\int_0^{\pi/2}x\textrm{d}x-\displaystyle\frac{\log(2)}{2}\underbrace{\int_0^{\pi/2}x\log(\sin(x))\textrm{d}x}_{\displaystyle 7/16 \zeta(3)-3/4\log(2)\zeta(2)}+\frac{1}{4}\int_0^{\pi/2}x \log ^2(\sin(x))\textrm{d}x$$ and $$\int_0^{\pi/4} x\log^2(\cos(x)) \textrm{d}x= \int_{\pi/4}^{\pi/2} \left(\frac{\pi}{2}-x\right) \log^2(\sin(x)) \textrm{d}x$$ $$=\int_{0}^{\pi/2} \left(\frac{\pi}{2}-x\right) \log^2(\sin(x)) \textrm{d}x-\int_{0}^{\pi/4} \left(\frac{\pi}{2}-x\right) \log^2(\sin(x)) \textrm{d}x$$ $$=\frac{\pi}{2}\underbrace{\int_{0}^{\pi/2}\log ^2(\sin(x))\textrm{d}x}_{\displaystyle \pi^3/24+\pi/2 \log^2(2)}-\int_{0}^{\pi/2}x\log ^2(\sin(x))\textrm{d}x-\frac{\pi}{2}\int_{0}^{\pi/4}\log ^2(\sin(x))\textrm{d}x$$ $$+\int_{0}^{\pi/4}x\log ^2(\sin(x))\textrm{d}x,$$

based on $(2)$, we obtain that

$$\int_0^{\pi/4} x \log ^2(\sin(x)) \textrm{d}x$$ $$=-\frac{15 }{16}\zeta(4)-\frac{39}{64}\log^2(2)\zeta (2)-\frac{7}{128} \log(2)\zeta(3)+\frac{\pi}{4}\int_0^{\pi/4} \log ^2(\sin (x)) \textrm{d}x$$ $$+\frac{9}{16}\int_0^{\pi/2} x \log ^2(\sin (x)) \textrm{d}x+\frac{1}{4} \int_0^{\pi/4} x \log^2(\tan(x)) \textrm{d}x$$ $$=\frac{1}{8}\log (2)\pi G+\frac{5}{384}\log ^4(2)+\frac{5}{32}\log^2(2)\zeta(2)-\frac{35}{128}\log(2)\zeta(3)+\frac{95}{256}\zeta(4)$$ $$-\frac{\pi}{4}\Im\biggr\{\operatorname{Li}_3\left(\frac{1+i}{2}\right)\biggr\}+\frac{5 }{16}\operatorname{Li}_4\left(\frac{1}{2}\right),$$

which is the desired closed-form.

In the calculations we also used that

$$\int_0^{\pi/4} \log^2(\sin (x))\textrm{d}x=\frac{23}{384}\pi^3+\frac9{32}\pi\log^2(2)+\frac{1}{2}\log(2)G-\Im\biggr\{\operatorname{Li}_3\left(\frac{1+i}{2}\right)\biggr\},$$ which can be extracted from this answer, next

$$\int_0^{\pi/2} x \log ^2(\sin (x))\textrm{d}x=\operatorname{Li}_4\left(\frac{1}{2}\right)+\frac{1}{24}\log^4(2)+\frac{1}{2}\log^2(2)\zeta(2)-\frac{19}{32}\zeta(4),$$ which is proved here and here, and finally $$\int_0^{\pi/4} x \log^2(\tan(x)) \textrm{d}x=\int_0^1 \frac{\arctan(x)}{1+x^2} \log^2(x) \textrm{d}x$$ $$=\frac{151}{128}\zeta(4)+\frac{1}{4}\log^2(2)\zeta(2)-\frac{7}{8}\log(2)\zeta(3)-\frac{1}{24}\log^4(2)-\operatorname{Li}_4\left(\frac{1}{2}\right),$$

which can be extracted by using a key advanced series from this answer, beautifully derived with the help of a Fourier series from the book, (Almost) Impossible Integrals, Sums, and Series.

End of (beautiful) story

A note: the solution nicely and completely avoids the necessity of using contour integration.


Thanks for the input, Fred. I managed to evaluate these types of integrals by using

$\displaystyle \ln(1-e^{ix})=\ln(2\sin(x/2))-i/2(\pi-x)$. Mathematica often gives

hypergeometric solutions, which I have no interest in. These log-sin integrals can be done,

but they are extremely onerous. I managed to work out the above integral except the upper limit was $\frac{\pi}{2}$.

It was extremely tedious, but does have a closed form in terms of

polylogs, zeta, and so on. As does the $\frac{\pi}{4}$ case.

For instance, evaluating $\displaystyle \int_{0}^{\frac{\pi}{2}}x\ln^{2}(2\sin(x/2))dx$ gives

$\displaystyle \frac{5}{96}\ln^{4}(2)-\frac{{\pi}^{2}\ln^{2}(2)}{48}+\frac{5}{4}Li_{4}(1/2)+\frac{19{\pi}^{4}}{1152}+\pi\mathfrak{I} \left[Li_{3}\left(1/2+i/2\right)\right]+\frac{35}{32}\zeta(3)\ln(2)-\frac{\pi\ln(2)}{2}G$.

This required a lot of patience and work on my part to arrive at this. The $\frac{\pi}{4}$ case is even worse to evaluate.

But, regarding the math engine result: take the indefinite integral

$\displaystyle \int\frac{1-x}{(1+x)\sqrt{x^{3}+x^{2}+x}}dx$.

This has a rather simple closed form of

$\displaystyle-\sin^{-1}\left(\frac{x^{2}+1}{(x+1)^{2}}\right)-\frac{\pi}{2}$,

but all I could get Mathematica and Maple to return was a long, complicated EllipticF mess.