Solution 1:

Suppose WLOG $b > a$. Then with $c = b/a$ we have

$$\begin{align} I &= \int_0^\pi \log(a^2 + b^2 - 2ab \cos x) \, dx \\ &=\int_0^\pi \log a^2 \, dx + \int_0^\pi \log(1 + (b/a)^2 - 2(b/a) \cos x) \, dx \\ &= 2\pi \log a + \int_0^\pi \log(1 + c^2 - 2c \cos x) \, dx \end{align}$$

Now we can evaluate the second integral on the RHS as the limit of a Riemann sum

$$\begin{align}J &= \int_0^\pi \log(1 + c^2 - 2c \cos x) \, dx \\ &= \lim_{n \to \infty} \frac{\pi}{n}\sum_{j=1}^n\log(1 + c^2 - 2c\cos(\pi(j-1)/n)) \\ &= \lim_{n \to \infty} \frac{\pi}{n}\log\prod_{j=1}^n(1 + c^2 - 2c\cos(\pi(j-1)/n)) \\ &=\lim_{n \to \infty} \frac{\pi}{n}\log(1-c)^2+\lim_{n \to \infty} \frac{\pi}{n}\log\prod_{j=2}^{n}(1 + c^2 - 2c\cos(\pi(j-1)/n)) \\ &= \lim_{n \to \infty} \frac{\pi}{n}\log\prod_{j=1}^{n-1}(1 + c^2 - 2c\cos(\pi j/n))\end{align} .$$

Upon factoring, we obtain

$$1 + c^2 - 2c\cos(\pi j/n) = [c - \exp(i \pi j /n)][c - \exp(-i \pi j /n)],$$

and

$$c^{2n} -1 = (c-1)(c+1)\prod_{j=1}^{n-1}[c - \exp(i \pi j /n)][c - \exp(-i \pi j /n)].$$

Hence, recalling that $c = b/a > 1$, it follows that

$$\begin{align}J &= \lim_{n \to \infty} \frac{\pi}{n} \log\left(\frac{c^{2n}-1}{c^2-1} \right) \\ &= \pi \lim_{n \to \infty} \log\left(\frac{c^{2n}-1}{c^2-1} \right)^{1/n} \\ &= \pi \lim_{n \to \infty} \log\left[c^2 \left(\frac{1-c^{-2n}}{c^2-1} \right)^{1/n}\right] \\ &= \pi \log c^2 + \log \left[\lim_{n \to \infty} \left(\frac{1-c^{-2n}}{c^2-1} \right)^{1/n}\right]\\ &= \pi \log c^2\end{align}$$

Thus,

$$I = 2\pi \log a + \pi \log c^2 = 2\pi \log a + \pi \log (b/a)^2 \\ = 2\pi \log b $$

Solution 2:

Suppose that $b>a$. Then $$ \int_{0}^{\pi}\log(a^{2}+b^{2}-2ab\cos x)\, dx = 2\pi\log b + \int_{0}^{\pi}\log(c^{2}+ 1-2c\cos x)\, dx $$ where $c = \dfrac{a}{b}$ and $0 \le c < 1$.

We will prove that $$ I = \int_{0}^{\pi}\log(c^{2}+ 1-2c\cos x)\, dx = 0. $$ If $\log$ is the principal branch then $f(z) = \dfrac{\log(1-z)}{z}$ is an analytic function in |z| <1. Consequently $$ 0 = \oint_{|z|=c}f(z)\, dz = \int_{-\pi}^{\pi}\dfrac{\log(1-ce^{it})}{ce^{it}}ice^{it}\, dt = i\int_{-\pi}^{\pi}\log(1-ce^{it})\, dt $$ The imaginary part of the right hand side is $$ 0 = \int_{-\pi}^{\pi}\log|1-ce^{it}|\, dt = \int_{-\pi}^{\pi}\log(\sqrt{c^{2}+1-2c\cos t})\, dt = \int_{0}^{\pi}\log(c^{2}+1-2c\cos t)\, dt = I. $$