Formula for $\int_0^\infty \frac{\log(1+x^2)}{\sqrt{(a^2+x^2)(b^2+x^2)}}dx$
Solution 1:
Assume $a > b$, define $m$ and $s$ such that $\frac{b^2}{a^2} = 1 - m = \frac{1-3s}{2}$. Substitute $x^2$ by $\frac{b^2}{p-\frac{1-s}{2}}$, we can rewrite $I(a,b)$ as
$$I(a,b) = -\frac{1}{a}\int_\infty^{\frac{1-s}{2}} \frac{\log\left[\frac{p-\left(\frac{1-s}{2}-b^2\right)}{p-\left(\frac{1-s}{2}\right)}\right]}{ \sqrt{4p^3 - g_2 p - g_3}} dz \quad\text{ where }\quad \begin{cases}g_2 = 1+3s^2\\g_3 = s(s^2-1)\end{cases}$$ Let $\wp(z), \zeta(z), \sigma(z)$ be the Weierstrass elliptic, zeta and sigma functions associated with the ODE:
$$\wp'(z)^2 = 4\wp(z)^3 - g_2 \wp(z) - g_3 = 4 \left(\wp(z) - \frac{1-s}{2}\right)\left(\wp(z) + \frac{1+s}{2}\right)\left(\wp(z) - s\right)$$
The double poles of $\wp(z)$ lies on a rectangular lattice with half period
$$\omega = K(\sqrt{m}),\quad \omega' = iK'(\sqrt{m}) = iK(\sqrt{1-m})$$
where $K(k)$ is the complete elliptic integral of the first kind. Furthermore, $\wp(\omega) = \frac{1-s}{2}$ and $\wp(\omega') = -\frac{1+s}{2}$.
Choose $\rho$ such that $\wp(\omega \pm \rho ) = \frac{1-s}{2} - b^2$. When $b$ increases from $0$ to $\infty$, $\wp(\omega\pm\rho)$ decreases from $\wp(\omega) = \frac{1-s}{2}$ to $-\infty$. For concreteness, we will choose $\rho$ such that as $b$ varies from $0$ to $\infty$. $\omega + \rho$ will move along the polygonal path joining $\omega, \omega+\omega', \omega'$ and $0$. i.e. for small $b$, $\rho$ is purely imaginary.
In terms of all these, we can simplify $I(a,b)$ as
$$\begin{align} I(a,b) &= \frac{1}{a}\int_0^\omega\log\left[ \frac{\wp(z)-\wp(\omega+\rho)}{\wp(z)-\wp(\omega)} \right]dz = \frac{1}{2a}\int_{-\omega}^\omega\log\left[ \frac{\wp(z)-\wp(\omega+\rho)}{\wp(z)-\wp(\omega)}\right]dz\\ &= \frac{1}{2a}\int_{-\omega}^\omega\log\left[C \frac{\sigma(z+\omega+\rho)\sigma(z+\omega-\rho)}{\sigma(z+\omega)^2} \right] \end{align} $$ where $\displaystyle\;C = \frac{\sigma(\omega)^2}{\sigma(\omega+\rho)\sigma(\omega-\rho)}$.
Using the same tricks as this answer, we can evaluate the $z$ dependent part and get $$\begin{align} I(a,b) &= \frac{1}{2a}\left( 2\omega\log C + \varphi_{+}(\omega+\rho) + \varphi_{-}(\omega-\rho)-\varphi_{+}(\omega)-\varphi_{-}(\omega)\right)\\ &= \frac{1}{2a}\left( 2\omega\log C + \zeta(\omega)((\omega+\rho)^2 + (\omega-\rho)^2 - 2\omega^2) -\pi i((\omega + \rho) - (\omega - \rho))\right)\\ &= \frac{1}{a}\left( \omega \log C + \zeta(\omega) \rho^2 - \pi i \rho\right) \end{align}$$ Using various identities for the elliptic functions: $$\begin{align} \wp(z) - \wp(u) &= -\frac{\sigma(z+u)\sigma(z-u)}{\sigma(z)^2\sigma(u)^2}\\ \wp(\omega) + \wp(\rho) + \wp(\omega+\rho) &= \frac14\left(\frac{\wp'(\omega)-\wp'(\omega+\rho)}{\wp(\omega)-\wp(\omega+\rho)}\right)^2 \end{align} $$ We find $$C = -\frac{1}{\sigma(\rho)^2(\wp(\omega)-\wp(\rho))} \quad\text{ and }\quad \wp(\omega) - \wp(\rho) = \frac{1}{a^2}$$ This finally gives us
$$I(a,b) = \frac{1}{a}\left\{ \omega\log\left[ -\frac{a^2}{\sigma(\rho)^2}\right] + \zeta(\omega) \rho^2 - \pi i\rho \right\} \quad\text{ with }\quad \rho = \wp^{-1}\left(\frac{a^2+b^2-3}{3a^2}\right) $$ As a double check, let us study two special cases $b = 1$ and $a = 1$.
Case I: $b = 1$.
When $b = 1$, $\wp(\rho) = s = \wp(\omega'\pm\omega) \implies \rho = \omega'-\omega$. Let $\eta = \zeta(\omega)$ and $\eta' = \zeta(\omega')$. The addition formula for sigma function $$\wp'(z+\rho) = -\frac{\sigma(2z+2\rho)}{\sigma(z+\rho)^4} = e^{2(\eta'-\eta)(2z + \omega'-\omega)}\frac{\sigma(2z)}{\sigma(z+\rho)^4}$$ implies $$\sigma(\rho)^4 = \frac{2}{\wp''(\rho)}e^{2(\eta'-\eta)(\omega'-\omega)} = -\frac{a^4}{b^2(a^2-b^2)}e^{2(\eta'-\eta)(\omega'-\omega)} $$ Up to a sign factor in the intermediate imaginary pieces, this allow us to fix $I(a,b)$ to the form
$$\frac{1}{a}\left\{\omega\left(\log(b\sqrt{a^2-b^2}) \stackrel{?}{\pm} \frac{\pi i}{2}\right) -\omega ( (\eta'-\eta)(\omega'-\omega) ) + \eta(\omega'-\omega)^2 - \pi i(\omega'-\omega) \right\}$$
Using the identity $\eta\omega'-\omega\eta' = \frac{\pi i}{2}$, the mess after the logarithm can be simplified: $$\begin{align} & -\omega ( (\eta'-\eta)(\omega'-\omega) ) + \eta(\omega'-\omega)^2 - \pi i(\omega'-\omega)\\ =& (\omega'-\omega)( -\omega(\eta'-\eta) + \eta(\omega'-\omega) - \pi i)\\ =& -\frac{\pi i}{2}(\omega' - \omega) \end{align} $$ We know $I(a,b)$ is a real number, the imaginary part $\frac{\pi i}{2}\omega$ in above expression will get cancelled by the imaginary piece associated with the logarithm. As a result, we find when $b = 1$, $$\begin{align} I(a,1) &= \frac{1}{a}\left\{\omega\log(b\sqrt{a^2-b^2}) - \frac{\pi i}{2}\omega'\right\}\\ &= \frac{1}{a}\left\{\log(b\sqrt{a^2-b^2})K\left(\sqrt{1-\frac{b^2}{a^2}}\right) + \frac{\pi}{2}K\left(\frac{b}{a}\right)\right\} \end{align}$$ reproducing what is known.
Case II: $a = 1$.
When $a = 1$, $\wp(\rho) = -\frac{1+s}{2} = \wp(\pm\omega') \implies \rho = \omega'$. Once again, by the addition formula for sigma function, we have $$\sigma(\rho)^4 = \frac{2}{\wp''(\rho)}e^{2\eta'\omega'} = \frac{a^2}{a^2-b^2}e^{2\eta'\omega'}$$ This allow us to conclude $I(a,b)$ has the form $$\frac{1}{a}\left\{\omega\log(a\sqrt{a^2-b^2}) - \omega\eta'\omega' + \eta\omega'^2 - \pi i\omega'\right\} = \frac{1}{a}\left\{\omega\log(a\sqrt{a^2-b^2}) -\frac{\pi i}{2}\omega'\right\} $$ and hence $I(1,b)$ can be casted to a form very similar to what we have for $b = 1$. $$I(1,b) = \frac{1}{a}\left\{\log(a\sqrt{a^2-b^2})K\left(\sqrt{1-\frac{b^2}{a^2}}\right) + \frac{\pi}{2}K\left(\frac{b}{a}\right)\right\} $$