Using residues to evaluate the integral $\int_{-\pi}^{\pi} \frac{\cos(n\theta)}{1-2a\cos(\theta)+a^2}d\theta$, $|a|<1$

Look at the integral $$ I_n = \int_{-\pi}^{\pi} \frac{e^{in\theta}}{(1-ae^{i\theta})(1-ae^{-i\theta})} \, d\theta. $$ It is easy to see the denominator expands to $1-2a\cos{\theta}+a^2$.

Since the interval of integration is symmetric, we can add the integral with $\theta \mapsto -\theta$ to $I$, and we find the integrand is $$ \frac{2\cos{n\theta}}{1-2a\cos{\theta}+a^2}, $$ so $I_n$ is equal to your integral. Now set $e^{i\theta}=z$, so the interval of integration transforms to the circle $|z|=1$, and $dz/z = i d\theta$. Hence $$ I_n = \frac{1}{i}\int_{|z|=1} \frac{z^{n}}{(1-az)(z-a)} \, dz. $$ Now you do the residues bit: since $|a|<1$, the only pole inside the unit circle is at $z=a$, where the residue is just $$ \left. \frac{z^n}{1-az} \right\rvert_{z=a} = \frac{a^n}{1-a^2} $$ Now we do $I_n = 2\pi i \sum \text{Res}$, and so $$ I_n = \frac{2\pi i}{i} \frac{a^n}{1-a^2} = \frac{2\pi a^n}{1-a^2}. $$

use the unit circle as the contour, so that we have $z=e^{i\theta}$ with $d\theta = \frac{dz}{iz}$

since $\cos n\theta = \Re z^n$ we require $\Re I$ where:

$$ I = \int_{|z|=1} \frac{z^n}{1-a(z+\frac1{z})+a^2}\frac{dz}{iz} \\ $$ $$ = -\frac1{ia}\int_{|z|=1} \frac{z^n}{z^2-(a+\frac1{a})z+1}dz $$

$$ = -\frac1{ia}\int_{|z|=1} \frac{z^n}{(z-a)(z-\frac1{a})}dz $$ since $|a| \lt 1$ only the residue at this point is required which is easily seen to be $\frac{a^n}{a-\frac1{a}}$

hence $$ I = -\frac1{ia} 2\pi i \frac{a^n}{a-\frac1{a}} = 2\pi \frac{a^n}{1-a^2} $$ since $I$ is already real, this is the required value of the original integral