Prove $\int_0^1 \frac{\tanh^{-1} (\beta t) dt}{t\sqrt{(1-t)(1- \alpha t)}}=\log (a) \log (b)$

We can solve this integral using only substitutions and integration by parts, as follows: $$I:=\int_0^1 \frac{\operatorname{arctanh} (\beta t) }{t\sqrt{(1-t)(1- \alpha t)}}dt=\int_0^1 \frac{\operatorname{arctanh}(\beta t)}{t(1-t)}\sqrt{\frac{1-t}{1-\alpha t}}dt$$ $$\overset{\large \frac{1-t}{1-\alpha t}=x}=\int_0^1 \frac{\operatorname{arctanh}\left(\beta \frac{1-x}{1-\alpha x}\right)}{\sqrt x(1-x)}dx\overset{x=y^2}=2\int_0^1 \frac{\operatorname{arctanh}\left(\beta \frac{1-y^2}{1-\alpha y^2}\right)}{1-y^2}dy$$ $$\overset{\large y=\frac{1-x}{1+x}}=\int_0^1 \operatorname{arctanh}\left( \frac{4\beta x}{(1+x)^2-\alpha (1-x)^2}\right)\frac{dx}{x}=\frac12 \int_0^1 \ln\left(\frac{\left(ab+x\right)\left(\frac{1}{ab}+x\right)}{\left(\frac{a}{b}+x\right)\left(\frac{b}{a}+x\right)}\right)\frac{dx}{x}$$ $$\overset{IBP}=\frac12 \int_0^1 \ln x \left(\frac{1}{\frac{a}{b}+x}+\frac{1}{\frac{b}{a}+x}-\frac{1}{ab+x}-\frac{1}{\frac{1}{ab}+x}\right)dx$$

In each of the integral from above we will simplify the denominator using the substitution $x\to kx$, where $k$ is the constant found in each denominator.

$$\Rightarrow I=\frac12 \left(\int_0^\frac{b}{a}\frac{\ln\left(\frac{a}{b}x\right)}{1+x}dx+\int_0^\frac{a}{b}\frac{\ln\left(\frac{b}{a}x\right)}{1+x}dx-\int_0^\frac{1}{ab}\frac{\ln\left(ab x\right)}{1+x}dx-\int_0^{ab}\frac{\ln\left(\frac{x}{ab}\right)}{1+x}dx\right)$$ $$\small =\color{red}{\frac12} \left(\ln\left(\frac{a}{b}\right)\ln\left(1+\frac{b}{a}\right)+\ln\left(\frac{b}{a}\right)\ln\left(1+\frac{a}{b}\right)-\ln(ab)\ln\left(1+\frac{1}{ab}\right)-\ln\left(\frac{1}{ab}\right)\ln\left(1+ab\right)\right)$$ $$+\color{chocolate}{\frac12}\left(\int_0^\frac{b}{a}\frac{\ln x}{1+x}dx+\int_0^\frac{a}{b}\frac{\ln x}{1+x}dx-\int_0^\frac{1}{ab}\frac{\ln x}{1+x}dx-\int_0^{ab}\frac{\ln x}{1+x}dx\right)$$ We can also rewrite the four integrals from above as: $$\color{blue}{\int_\frac{1}{ab}^\frac{b}{a}\frac{\ln x}{1+x}dx}+\int_{ab}^\frac{a}{b}\frac{\ln x}{1+x}dx\overset{\color{blue}{x\to \frac{1}{x}}}=\color{blue}{\int_{ab}^\frac{a}{b}\frac{\ln x}{x}dx-\int_{ab}^\frac{a}{b}\frac{\ln x}{1+x}dx}+\int_{ab}^\frac{a}{b}\frac{\ln x}{1+x}dx$$ $$=\int_{ab}^\frac{a}{b}\frac{\ln x}{x}dx=\frac{\ln^2 x}{2}\bigg|_{ab}^\frac{a}{b}=-2\ln a\ln b$$ So with some algebra for the first term we finally get: $$I=\color{red}{\frac12}\left(4\ln a \ln b\right)+\color{chocolate}{\frac12}\left(-2\ln a \ln b\right)=\boxed{\ln a\ln b}$$

An alternative approach using Feynman's trick can be found here, which shows: $$\int_0^1 \ln\left(\frac{\left(ab+x\right)\left(\frac{1}{ab}+x\right)}{\left(\frac{a}{b}+x\right)\left(\frac{b}{a}+x\right)}\right)\frac{dx}{x}=2\ln a\ln b$$ It might be useful in the future so I'll also mention that, since $\int_0^1 \frac{\ln x}{t+x}dx=\operatorname{Li}_2\left(-\frac{1}{t}\right)$ the following Dilogarithm identity arises from above: $$\boxed{\operatorname{Li}_2\left(-\frac{a}{b}\right)+\operatorname{Li}_2\left(-\frac{b}{a}\right)-\operatorname{Li}_2\left(-ab\right)-\operatorname{Li}_2\left(-\frac{1}{ab}\right)=2\ln a\ln b;\ a,b>0}$$

The given integral, after the substitution $t=\dfrac{(1-\alpha)x}{1-\alpha x}$, is equal to $$\int_0^1\tanh^{-1}\frac{\beta x}{1-\alpha+\alpha x}\frac{dx}{x\sqrt{1-x}}=\frac{1}{2}\Bigg(f\Big(\underbrace{\frac{\alpha+\beta}{1-\alpha}}_{=\frac{(ab-1)^2}{4ab}}\Big)-f\Big(\underbrace{\frac{\alpha-\beta}{1-\alpha}}_{=\frac{(a-b)^2}{4ab}}\Big)\Bigg),$$ where $$f(a)=\int_0^1\frac{\log(1+ax)}{x\sqrt{1-x}}~dx=\begin{cases}\color{blue}{2\log^2(\sqrt{a}+\sqrt{a+1})},&\phantom{-1\leqslant{}}a\geqslant0\\-2\arcsin^2\sqrt{-a},&-1\leqslant a<0\end{cases}$$ can be evaluated in many ways; a simple one is via "Feynman's trick": for $a>0$, $$f'(a)=\int_0^1\frac{dx}{(1+ax)\underbrace{\sqrt{1-x}}_{=y}}=2\int_0^1\frac{dy}{1+a-ay^2}\\=\frac{2}{\sqrt{a(a+1)}}\tanh^{-1}\sqrt\frac{a}{a+1}=2\frac{\log(\sqrt{a}+\sqrt{a+1})}{\sqrt{a(a+1)}}.$$

Probably there’s a more elegant and concise solution, but this is what I’ve got: $$I=\int_0^1 \underbrace{\frac{tanh^{-1}(\beta t)}{t\sqrt{(1-t)(1-\alpha t)}}dt}_{t\rightarrow\frac{1-t}{1+t}}=\sqrt{2}\int_0^1 \underbrace{\frac{tanh^{-1}(\beta \frac{1-t}{1+t})}{(1-t)\sqrt{(1+\alpha)t+1-\alpha}}\frac{dt}{\sqrt{t}}}_{\sqrt{t}\rightarrow t}$$

$$I=2\sqrt{2}\int_0^1 \underbrace{\frac{tanh^{-1}(\beta \frac{1-t^2}{1+t^2})}{(1-t^2)\sqrt{(1+\alpha)t^2+1-\alpha}}dt}_{t\rightarrow tanh(x)}=2\sqrt{2}\int_0^{\infty} \underbrace{\frac{tanh^{-1}\left(\frac{\beta}{2sinh^2(x)+1}\right)}{\sqrt{2sinh^2(x)+1-\alpha}}cosh(x)dx}_{sinh(x)\rightarrow x}$$

$$I=2\sqrt{2}\int_0^{\infty} \underbrace{\frac{tanh^{-1}\left(\frac{\beta}{2x^2+1}\right)}{\sqrt{2x^2+1-\alpha}}dx}_{x\rightarrow \sqrt{\frac{1-\alpha}{2}}x}=2\int_0^{\infty}\underbrace{\frac{tanh^{-1}\left(\frac{\beta}{(1-\alpha)x^2+1}\right)}{\sqrt{x^2+1}}dx}_{x\rightarrow sinh(z)}$$

$$\color{blue}{2tanh^{-1}\left(\frac{\beta}{(1-\alpha)x^2+1}\right)=log\left(\frac{x^2+\frac{1+\beta}{1-\alpha}}{x^2+\frac{1-\beta}{1-\alpha}}\right)=log\left(\frac{x^2+\frac{(1+ab)^2}{4ab}}{x^2+\frac{(a+b)^2}{4ab}}\right)}$$

$$I=\int_0^{\infty}log\left(\frac{sinh^2(z)+\frac{(1+ab)^2}{4ab}}{sinh^2(z)+\frac{(a+b)^2}{4ab}}\right)dz=\int_0^{\infty}\underbrace{log\left[\frac{\left(ab+e^{-2z}\right)\left(abe^{-2z}+1\right)}{\left(ae^{-2z}+b\right)\left(a+be^{-2z}\right)}\right]dz}_{e^{-2z}\rightarrow z}$$

$$I=\frac{1}{2}\int_{0}^{1}\log{\left[\frac{z^2+\left(ab+\frac{1}{ab}\right)z+1}{z^2+\left(\frac{a}{b}+\frac{b}{a}\right)z+1}\right]}\frac{dz}{z}\overbrace{=}^{IBP}$$

$$I=\frac{1}{2}\int_{0}^{1}{\log{\left(z\right)}\left[\frac{2z+\left(\frac{a}{b}+\frac{b}{a}\right)}{z^2+\left(\frac{a}{b}+\frac{b}{a}\right)z+1}-\frac{2z+\left(ab+\frac{1}{ab}\right)}{z^2+\left(ab+\frac{1}{ab}\right)z+1}\right]dz}$$ Notice that due to symmetry $\int_0^1 g(z)dz=\frac{1}{2}\int_0^\infty g(z)dz$. Therefore:

$$I=\frac{1}{4}\int_{0}^{\infty}{\log{\left(z\right)}\left[\frac{2z+\left(\frac{a}{b}+\frac{b}{a}\right)}{z^2+\left(\frac{a}{b}+\frac{b}{a}\right)z+1}-\frac{2z+\left(ab+\frac{1}{ab}\right)}{z^2+\left(ab+\frac{1}{ab}\right)z+1}\right]dz}$$

To evaluate this integral let's use Complex Analysis. First, let's consider a keyhole contour and the following function:

$$f(z)=\log^2{\left(z\right)}\left[\frac{2z+\left(\frac{a}{b}+\frac{b}{a}\right)}{z^2+\left(\frac{a}{b}+\frac{b}{a}\right)z+1}-\frac{2z+\left(ab+\frac{1}{ab}\right)}{z^2+\left(ab+\frac{1}{ab}\right)z+1}\right]$$

$$\oint f(z)dz=\left(\int_\Gamma+\int_\gamma+\int_{ir}^{R+ir}+\int_{R-ir}^{-ir}\right)f(z)dz$$

Applying the ML Inequality it's easy to show that $\int_\Gamma f(z)dz \rightarrow 0$ as $R \rightarrow \infty$ and $\int_\gamma f(z)dz \rightarrow 0$ as $r \rightarrow 0$. The other two integrals from the RHS can be rewritten as:

$$\int_0^\infty [log^2{\left(z\right)}-(log(z)+2\pi i)^2]\left[\frac{2z+\left(\frac{a}{b}+\frac{b}{a}\right)}{z^2+\left(\frac{a}{b}+\frac{b}{a}\right)z+1}-\frac{2z+\left(ab+\frac{1}{ab}\right)}{z^2+\left(ab+\frac{1}{ab}\right)z+1}\right]dz$$

$$\int_0^\infty (-4\pi ilog(z)+4\pi^2)\left[\frac{2z+\left(\frac{a}{b}+\frac{b}{a}\right)}{z^2+\left(\frac{a}{b}+\frac{b}{a}\right)z+1}-\frac{2z+\left(ab+\frac{1}{ab}\right)}{z^2+\left(ab+\frac{1}{ab}\right)z+1}\right]dz$$

$$-4\pi i\int_0^\infty log(z)\left[\frac{2z+\left(\frac{a}{b}+\frac{b}{a}\right)}{z^2+\left(\frac{a}{b}+\frac{b}{a}\right)z+1}-\frac{2z+\left(ab+\frac{1}{ab}\right)}{z^2+\left(ab+\frac{1}{ab}\right)z+1}\right]dz+\color{red}{0}$$

$$\color{red}{4\pi^2\int_0^\infty \left[\frac{2z+\left(\frac{a}{b}+\frac{b}{a}\right)}{z^2+\left(\frac{a}{b}+\frac{b}{a}\right)z+1}-\frac{2z+\left(ab+\frac{1}{ab}\right)}{z^2+\left(ab+\frac{1}{ab}\right)z+1}\right]dz=4\pi^2log\left[\frac{z^2+\left(ab+\frac{1}{ab}\right)z+1}{z^2+\left(\frac{a}{b}+\frac{b}{a}\right)z+1}\right]_0^\infty}$$

Computing the residues: $$\oint f(z)dz=2\pi i\left[log^2\left(-\frac{a}{b}\right)+log^2\left(-\frac{b}{a}\right)-log^2\left(-ab\right)-log^2\left(-\frac{1}{ab}\right)\right]=-16\pi\ i log(a)log(b)$$

Hence, gathering the results: $$\int_0^\infty log(z)\left[\frac{2z+\left(\frac{a}{b}+\frac{b}{a}\right)}{z^2+\left(\frac{a}{b}+\frac{b}{a}\right)z+1}-\frac{2z+\left(ab+\frac{1}{ab}\right)}{z^2+\left(ab+\frac{1}{ab}\right)z+1}\right]dz=4log(a)log(b)$$

From this is possible to conclude that: $$I=\int_0^1\frac{tanh^{-1}(\beta t)}{t\sqrt{(1-t)(1-\alpha t)}}dt=\frac{1}{4}\int_{0}^{\infty}{\log{\left(z\right)}\left[\frac{2z+\left(\frac{a}{b}+\frac{b}{a}\right)}{z^2+\left(\frac{a}{b}+\frac{b}{a}\right)z+1}-\frac{2z+\left(ab+\frac{1}{ab}\right)}{z^2+\left(ab+\frac{1}{ab}\right)z+1}\right]dz}=log(a)log(b)$$