Evaluating $\int_a^b \arccos\left(x\,/\sqrt{(a+b)x-ab\,}\,\right)\,\mathrm {d}x$ assuming $0<a<b$

The answer is $$\int_{x=a}^b \arccos\left({x\over\sqrt{(a+b)x-ab}}\right)\,dx={\pi (b-a)^2 \over 4(a+b)}.$$

This is problem 11457 from the American Mathematical Monthly. The problem was published in the October 2009 issue, and the solution appeared in the April 2011 issue.

Using integration by parts $\int u \,dv=uv-\int v\,du$ with $u=\arccos\left({x\over\sqrt{(a+b)x-ab}}\right)$ and $v=x-ab/(a+b)$ we have $$\int_{a}^b \arccos\left({x\over\sqrt{(a+b)x-ab}}\right)\,dx =\int_a^b {(b+a)x-2ab\over 2(a+b)\sqrt{(b-x)(x-a)}}\,dx .\tag1 $$ Using the change of variables $w=(2x-(a+b))/(b-a)$, the integral on the right hand side of (1) becomes $$\int_{-1}^1{(b^2-a^2)w+(b-a)^2 \over 4(a+b)\sqrt{1-w^2}}\,dw ={ (b-a)^2 \over 4(a+b) } \int_{-1}^1{dw \over \sqrt{1-w^2}} ={\pi (b-a)^2 \over 4(a+b)},$$ since the integral of the odd function $w/\sqrt{1-w^2}$ over the interval $(-1,1)$ is zero, and $\int_{-1}^1 (1-w^2)^{-1/2}\,dw=\arcsin(1)-\arcsin(-1)=\pi$.

The integrand is the angle $\theta(x)$ in this diagram. I tried to find a geometric explanation for the OP's integral, but I failed. Maybe someone else can succeed?

A succinct and beautiful solution has already posted while I was writing this answer, so I'm not sure if it will be helpful...

Let $\alpha = \sqrt{b/a} > 1$ and $\beta = \frac{1}{2}(\alpha - \alpha^{-1})$. Then by the substitution $ x = \sqrt{ab} u$, it suffices to show that

$$\int_{\alpha^{-1}}^{\alpha} \arccos \left( \frac{u}{\sqrt{(\alpha + \alpha^{-1}) u - 1}} \right) \; du = \frac{\pi \beta^2}{\alpha + \alpha^{-1}}.$$

Now by the substitution $v = \sqrt{(\alpha + \alpha^{-1}) u - 1}$, it is equivalent to

$$\int_{\alpha^{-1}}^{\alpha} 2 v \arccos \left( \frac{v + v^{-1}}{\alpha + \alpha^{-1}} \right) \; dv = \pi \beta^2 .$$

Now performing integration by parts, the left hand side becomes

$$\int_{\alpha^{-1}}^{\alpha} \frac{v^2 - 1}{\sqrt{(\alpha + \alpha^{-1})^2 - (v + v^{-1})^2}} \; dv = \int_{\alpha^{-1}}^{\alpha} \frac{v^2 - 1}{\sqrt{(\alpha - \alpha^{-1})^2 - (v - v^{-1})^2}} \; dv .$$

Finally, plug $v = w + \sqrt{1+w^2}$. Then this integral reduces to

$$\int_{-\beta}^{\beta} \frac{w\left(w+\sqrt{1+w^2}\right)^2}{\sqrt{\beta^2 - w^2}\sqrt{1+w^2}} \; dw .$$

Now expanding the square in the numerator, the odd terms cancel out, yielding

$$\int_{-\beta}^{\beta} \frac{2w^2}{\sqrt{\beta^2 - w^2}} \; dw .$$

This integral is easily calculated by the substitution $w = \beta \sin\theta$, verifying the identity as desired.