Integral $\int_0^{\pi/4}\log \tan x \frac{\cos 2x}{1+\alpha^2\sin^2 2x}dx=-\frac{\pi}{4\alpha}\text{arcsinh}\alpha$

Integrate by parts; then you get that

$$I(\alpha) = \left [\frac1{2 \alpha} \arctan{(\alpha \sin{2 x})} \log{(\tan{x})} \right ]_0^{\pi/4} - \int_0^{\pi/4} dx \frac{\arctan{(\alpha \sin{2 x})}}{\alpha \sin{2 x}}$$

The first term on the RHS is zero. To evaluate the integral, expand the arctan into a Taylor series and get

$$I(\alpha) = -\frac12 \sum_{k=0}^{\infty} \frac{(-1)^k}{2 k+1} \alpha^{2 k} \int_0^{\pi/2} du \, \sin^{2 k}{u} = -\frac{\pi}{4} \sum_{k=0}^{\infty} \frac{(-1)^k}{2 k+1} \binom{2 k}{k} \left (\frac{\alpha}{2} \right )^{2 k}$$

A little manipulation leads us to

$$\alpha I'(\alpha) +I(\alpha) = -\frac{\pi}{4} \sum_{k=0}^{\infty} (-1)^k \binom{2 k}{k} \left (\frac{\alpha}{2} \right )^{2 k} = -\frac{\pi}{4} \frac1{\sqrt{1+\alpha^2}}$$

The LHS is just $[\alpha I(\alpha)]'$, so the solution is

$$I(\alpha) = -\frac{\pi}{4} \frac{\operatorname{arcsinh}(\alpha)}{\alpha} $$

My first step is similar to Ron Gordon but then I took a different route. From integration by parts, the given integral can be written as: $$-\frac{1}{a}\int_0^{\pi/4} \frac{\arctan(\alpha \sin(2x))}{\sin (2x)}\,dx$$ Consider $$I(a)=\int_0^{\pi/4} \frac{\arctan(a \sin(2x))}{\sin (2x)}\,dx$$ Differentiate both the sides wrt $a$ to obtain: $$I'(a)=\int_0^{\pi/4} \frac{1}{1+a^2\sin^2(2x)}\,dx$$ Use the substitution $a\sin(2x)=t$ to obtain: $$I'(a)=\frac{1}{2}\int_0^a \frac{dt}{\sqrt{a^2-t^2}(1+t^2)}$$ Next use the substitution $t=a/y$ to get: $$I'(a)=\frac{1}{2}\int_1^{\infty} \frac{y}{\sqrt{y^2-1}(a^2+t^2)}\,dy$$ With yet another substitution which is $y^2-1=u^2$, $$I'(a)=\frac{1}{2}\int_0^{\infty} \frac{du}{u^2+a^2+1}$$ The final integral is trivial, hence: $$I'(a)=\frac{\pi}{4\sqrt{a^2+1}}$$ Integrate both sides wrt $a$ to get: $$I(a)=\frac{\pi}{4}\sinh^{-1}a+C$$ It is easy to see that $C=0$, hence with $a=\alpha$, $$-\frac{1}{a}\int_0^{\pi/4} \frac{\arctan(\alpha \sin(2x))}{\sin (2x)}\,dx=-\frac{\pi}{4\alpha}\sinh^{-1}\alpha$$ $\blacksquare$

I use on Ron's results from integrating by parts:

$$\alpha I(\alpha) = - \int_0^{\pi/4} dx \frac{\arctan{(\alpha \sin{2 x})}}{\sin{2 x}},$$

As an alternative way to complete the problem, use the method differentiating under the integral sign:

$$\frac{d}{d\alpha}(\alpha I(\alpha)) = -\int_{0}^{\pi/4}\frac{dx}{\alpha^2\sin^2{2x}+1}=-\frac{\pi}{4\sqrt{1+\alpha^2}}\\ \implies \alpha I(\alpha) = -\frac{\pi}{4}\int_{0}^{\alpha}\frac{d\tilde\alpha}{\sqrt{1+\tilde\alpha^2}} = -\frac{\pi}{4}\sinh^{-1}{\alpha}.$$