About $\int_0^{\pi/2}\arctan(1-\sin^2 (x) \cos^2 (x))dx = \pi \left( \frac{\pi}{4}-\arctan \sqrt{\frac{\sqrt{2}-1}{2}}\right)$

In this question sos440 has mentioned about an integral that he computed:

$$\int_0^{\pi/2}\arctan(1-\sin^2 (x) \cos^2 (x))dx = \pi \left( \frac{\pi}{4}- \arctan \sqrt{\frac{\sqrt{2}-1}{2}}\right)$$

I would really like to know how this can be proved. I tried using differentiation under the integral sign on parmeter $a$:

$$\int_0^{\pi/2}\arctan(1-a\sin^2 (x) \cos^2 (x))dx $$

but it didn't really work. I would be really grateful if sos440 or somebody else can tell me the secret behind discovering this formula.

Thanks!

$$ \begin{aligned} \int_0^{\pi/2} \arctan\left(1-\cos^2x\sin^2x\right)\,dx & =\int_0^{\pi/2} \left(\frac{\pi}{2}-\arctan\left(\frac{1}{1-\sin^2x\cos^2x}\right)\right)\,dx \\ &=\frac{\pi^2}{4}-\int_0^{\pi/2} \arctan(\sin^2x)\,dx-\int_0^{\pi/2} \arctan(\cos^2x)\,dx \\ &=\frac{\pi^2}{4}-2\int_0^{\pi/2}\arctan(\cos^2x)\,dx \\ \end{aligned} $$

Consider

$$I(a)=\int_0^{\pi/2} \arctan(a\,\cos^2x)\,dx $$

$$ \begin{aligned} \Rightarrow I'(a)&=\int_0^{\pi/2} \frac{\cos^2x}{1+a^2\cos^4x}\,dx \\ &= \int_0^{\pi/2} \frac{\sec^2x}{\sec^4x+a^2}\,dx\\ &= \int_0^{\pi/2} \frac{\sec^2x}{(1+\tan^2x)^2+a^2}\,dx\\ &=\int_0^{\infty} \frac{dt}{t^4+2t^2+a^2+1}\,\,\,\,\,\,\,(\tan x=t) \\ \end{aligned} $$

With the substitution $t\mapsto \dfrac{\sqrt{a^2+1}}{t} $,

$$I'(a)=\frac{1}{\sqrt{a^2+1}}\int_0^{\infty} \frac{t^2}{t^4+2t^2+a^2+1}\,dt $$

$$ \begin{aligned} \Rightarrow I'(a) &=\frac{1}{2\sqrt{a^2+1}}\int_0^{\infty} \frac{\sqrt{a^2+1}+t^2}{t^4+2t^2+a^2+1}\,dt \\ &=\frac{1}{2\sqrt{a^2+1}}\int_0^{\infty} \frac{1+\frac{\sqrt{1+a^2}}{t^2}}{\left(t-\frac{\sqrt{a^2+1}}{t}\right)^2+2(1+\sqrt{a^2+1})}\,dt\\ &=\frac{1}{2\sqrt{a^2+1}}\int_{-\infty}^{\infty} \frac{dy}{y^2+2(1+\sqrt{a^2+1})}\,\,\,\,\,\,\,\left(t-\frac{\sqrt{a^2+1}}{t}=y\right) \\ &=\frac{\pi}{2\sqrt{2}} \frac{1}{\sqrt{1+a^2}\sqrt{1+\sqrt{a^2+1}}} \\ \end{aligned} $$

Integrating back,

$$ \begin{aligned} I(1)-I(0)=I(1) &=\frac{\pi}{2\sqrt{2}}\int_0^1 \frac{da}{\sqrt{1+a^2}\sqrt{1+\sqrt{a^2+1}}} \\ &= \frac{\pi}{2\sqrt{2}}\int_1^{\sqrt{2}} \frac{dt}{\sqrt{t-1}(t+1)}\,\,\,\,\,\,\,(\sqrt{a^2+1}=t) \\ &=\frac{\pi}{\sqrt{2}}\int_0^{\sqrt{\sqrt{2}-1}} \frac{du}{u^2+2} \,\,\,\,\,\,\,(t-1=u^2) \\ &=\frac{\pi}{2} \left(\arctan\frac{u}{\sqrt{2}}\right|_0^{\sqrt{\sqrt{2}-1}} \\ &=\frac{\pi}{2}\arctan\left(\sqrt{\frac{\sqrt{2}-1}{2}}\right) \\ \end{aligned}$$

Hence,

$$\boxed{\displaystyle \int_0^{\pi/2} \arctan\left(1-\cos^2x\sin^2x\right)\,dx=\dfrac{\pi^2}{4}-\pi\arctan\left(\sqrt{\dfrac{\sqrt{2}-1}{2}}\right)}$$

Updated (2019-08-19). Thanks to @darij grinberg I salvaged the content of my broken blog page. But I strongly recommend the reader to see @Pranav Arora's beautiful and neat computation first. This answer is more of 'brutal-force' solution.

Let $I$ denote this integral hereafter. To achieve this, we first observe by double-angle formula that

$$ \begin{aligned} I &= \int_{0}^{\frac{\pi}{2}} \arctan\left(1-\tfrac{1}{4}\sin^2 x\right) \, \mathrm{d}x \\ &= \frac{1}{2} \int_{0}^{\pi} \arctan\left(\tfrac{1}{8}(7+\cos x)\right) \, \mathrm{d}x \\ &=\frac{1}{4} \int_{-\pi}^{\pi} \arctan\left(\tfrac{1}{8}(7+\cos x)\right) \, \mathrm{d}x. \end{aligned} $$

In the first and second step, we used the half-angle substitution $2x\mapsto x$. Then by symmetry, this is written as

Then by adopting the identity

$$ \int_{1}^{\infty} \frac{\alpha}{\alpha^2+t^2} \, \mathrm{d}t = \arctan(\alpha) \tag{$\alpha > 0$}, $$

the last identity for $I$ can be recast to

$$ I = 2 \int_{1}^{\infty} \int_{-\pi}^{\pi} \frac{7+\cos x}{(7+\cos x)^2 + 64t^2} \, \mathrm{d}x\mathrm{d}t. $$

Then we make change of variable $z = e^{ix}$ to obtain

$$ I = \frac{4}{i} \int_{1}^{\infty} \int_{\mathcal{C}} \frac{z^2 + 14z + 1}{(z^2 + 14z + 1)^2 + (16tz)^2} \, \mathrm{d}z\mathrm{d}t, $$

where $\mathcal{C}$ denote the counter-clockwised unit circle centered at the origin. Now let $f(z)$ denote the integrand of the above equality. To apply Cauchy integration formula, we have to find the pole inside the circle $\mathcal{C}$. To this end, let $\alpha_{\pm}, \beta_{\pm}$ denote

$$ \begin{gathered} \alpha_{\pm} = -(7+8it) \pm\sqrt{(7+8it)^2-1}, \\ \beta_{\pm} = -(7-8it)\pm\sqrt{(7-8it)^2 - 1}, \end{gathered}$$

respectively. Note that $\alpha_{\pm}$ and $\beta_{\pm}$ are zeros of quadratic polynomials

$$z^2 + (14+16it)z + 1 \qquad \text{and} \qquad z^2 + (14-16it)z + 1$$

respectively, so that $f(z)$ factors into the form

$$ f(z) = \frac{z^2 + 14z + 1}{(z-\alpha_+)(z - \alpha_-)(z - \beta_+)(z - \beta_-)}. $$

Now for $t > 1$, only $\alpha_+$ and $\beta_+$ are contained in the inside of $\mathcal{C}$. Thus by Cauchy integration formula,

$$ \begin{aligned} I &= 8\pi \int_{1}^{\infty} \left( \underset{z=\alpha_+}{\mathrm{Res}} \, f(z) + \underset{z=\beta_+}{\mathrm{Res}} \, f(z) \right) \, \mathrm{d}t \\ &= 8\pi \int_{1}^{\infty} \left( \frac{1}{2(\alpha_+ - \alpha_-)} + \frac{1}{2(\beta_+ - \beta_-)} \right) \, \mathrm{d}t \\ &= 2\pi \int_{1}^{\infty} \left( \frac{1}{\sqrt{(7+8it)^2 - 1}} + \frac{1}{\sqrt{(7-8it)^2 - 1}} \right) \, \mathrm{d}t \end{aligned} $$

To evaluate the integral in the right hand side, we note that

$$ \frac{d}{dz} \log(z+\sqrt{z^2-1}) = \frac{1}{\sqrt{z^2 - 1}}. $$

on the right-half plane $\{ z \in \mathbb{C} : \operatorname{Re}(z) > 0\}$. Then we may proceed as

$$\begin{aligned} I &= \frac{\pi}{4i} \left[ \log\left(7+8it + \sqrt{(7+8it)^2 - 1}\right) - \log\left(7-8it + \sqrt{(7-8it)^2 - 1}\right) \right]_{1}^{\infty} \\ &= \frac{\pi}{2} \left[ \arg\left(7+8it + \sqrt{(7+8it)^2 - 1}\right) \right]_{1}^{\infty} \\ &= \frac{\pi}{2} \left( \frac{\pi}{2} - \arg \left( 7+8i + \sqrt{(7+8i)^2 - 1}\right) \right). \end{aligned} $$

This horrifying argument term can be simplified further by noting that

$$ \frac{1}{2}\arg \left( 7+8i + \sqrt{(7+8i)^2 - 1}\right) = \frac{1}{2}\arctan\left( \frac{\sqrt{8}}{7}\sqrt{5\sqrt{2}+1} \right) = \arctan\sqrt{\frac{\sqrt{2}-1}{2}}. $$