Closed form of $\mathscr{R}=\int_0^{\pi/2}\sin^2x\,\ln\big(\sin^2(\tan x)\big)\,\,dx$

Here's a slightly different approach. Using the power reduction formula and getting rid of the second exponent of the sine using the properties of the logarithm the integral becomes:

$$I=\int_0^{\frac{\pi}{2}} (1-\cos 2x) \log (\sin (\tan x))dx$$

Using the substitution $2x=u$ we have:

$$I = \frac{1}{2}\int_0^\pi (1-\cos u) \log\bigg(\sin \bigg(\tan \bigg(\frac{u}{2}\bigg)\bigg)\bigg)du$$

We have the integral ready for a Weierstrass substitution, after which it becomes:

$$I = \int_0^\infty\bigg(1-\frac{1-t^2}{1+t^2}\bigg) \log(\sin (t)) \frac{1}{1+t^2}dt$$

Or:

$$I = \int_0^\infty \frac{2t^2}{(1+t^2)^2} \log(\sin (t))dt$$

From now on I'll use $x$ again. The Fourier series of $\log (\sin x)$ is well known and it is:

$$\log(\sin x)= -\log 2 -\sum_{n=1}^\infty \frac{\cos(2nx)}{n}$$

So the integral, exchanging integration and summation, becomes:

$$I=-2\log 2 \int_0^\infty \frac{x^2}{(1+x^2)^2}dx-2\sum_{n=1}^\infty \frac{1}{n} \int_0^\infty \frac{x^2\cos(2nx)}{(1+x^2)^2}$$

These are both easy integrals from the point of view of residue calculus. The final result is:

$$I=-\frac{\pi \log 2}{2} -\frac{\pi}{2}\sum_{n=1}^\infty \frac{e^{-2n}}{n}+\pi \sum_{n=1}^\infty e^{-2n}$$

These sums can be evaluated using the geometric series and its integral. So we have:

$$I=-\frac{\pi \log 2}{2}+\frac{\pi}{e^2-1}-\frac{\pi}{2}(2-\log(e^2-1))$$

Simplifying:

$$I=\frac{\pi}{2}\log \bigg( \frac{e^2-1}{2} \bigg) +\pi\bigg(\frac{2-e^2}{e^2-1}\bigg)$$

The answer is $$ \mathscr{R}=\frac{\pi }{2} \left(\log \left(\frac{e^2-1}{2} \right)-\frac{2 \left(e^2-2\right)}{e^2-1}\right) $$ As Kirill proved we have $$\eqalign{ \mathscr{R}&=\frac{1}{2}\int_{-\infty}^\infty\frac{u^2}{(1+u^2)^2}\log(\sin^2u)du\cr &=\frac{1}{2}\int_{0}^\pi\left(\sum_{k\in\mathbb{Z}}\frac{(u+k\pi)^2}{(1+(u+k\pi)^2)^2}\right)\log(\sin^2u)du } $$ Now, the function $$ F(u)=\sum_{k\in\mathbb{Z}}\frac{(u+k\pi)^2}{(1+(u+k\pi)^2)^2} $$ is $\pi$-periodic and even function. It is not difficult to calculate its Fourier cosine coefficients $a_n$ such that $$ F(u)=\frac{a_0}{2}+\sum_{n=1}a_n\cos(2n u) $$ with, $$a_n=\frac{2}{\pi}\int_0^\pi F(u)\cos(2n u)du =\frac{2}{\pi}\int_{-\infty}^\infty \frac{u^2}{(1+u^2)^2}\cos(2n u)du= e^{-2 n} (1-2 n)$$ The last equality is obtained by a simple residue calculus.

On the other hand it is easy and well-known that $$ \log(\sin^2u)=-2\log 2-\sum_{n=1}^\infty\frac{2}{n}\cos(2nu) $$ So using Parseval's formula we get $$ \mathscr{R}=\frac{\pi}{2}\left(- \log 2-\frac{1}{2}\sum_{n=1}^\infty\frac{2}{n}e^{-2 n} (1-2 n)\right) $$ and this simplifies easily to the announced closed form.$\qquad\square$

Write $\tan x = u$ to get the integral $$ \int_0^\infty \frac{u^2\,du}{ (1+u^2)^2} \log(\sin^2 u) $$ and split over periods of length $\pi$ by writing $u=\pi k+s$, $0<s<\pi$ so that the integral is $$ 2\int_0^\pi \sum_{k\geq0} \frac{u^2}{(1+u^2)^2}\log\sin s\,ds, \qquad u = \pi k+s. $$ The sum can be done explicitly in terms of polygamma functions: $$ \frac{1}{2\pi}\Im\psi\left(\frac{i+s}{\pi}\right) + \frac{1}{2\pi^2}\Re\psi_1\left(\frac{i+s}{\pi}\right), $$ so the integral is equal to $$ \frac1\pi \int_0^\pi \log(\sin s)\left(\Im\psi\left(\frac{i+s}{\pi}\right) + \frac{1}{\pi}\Re\psi_1\left(\frac{i+s}{\pi}\right) \right)\,ds$$ which is numerically $$ \begin{array}{rl} -0.8254932940&1920795045&3494583393&8145490721&3051472153&0015143141\\ 0263463153&8662518683&0960012709&2734877933&9171668805&2198716476\\ 0581876961&2557665495&3473838059&8389072188&9187974995&4963384740\\ 9429563810&6463831818&4148444098&6729534027&3239373746&3514130065&\ldots \end{array} $$