closed form of $\int_{0}^{2\pi}\frac{dx}{(a^2\cos^2x+b^2\sin^2x)^n}$

In the following we assume that $n\geq 1$.

Thanks to parity, the integral can be written as \begin{align} I_n=\int_0^{2\pi}\frac{dx}{(a^2\cos^2x+b^2\sin^2 x)^n}&=2\int_0^{\pi}\frac{dx}{(a^2\cos^2x+b^2\sin^2 x)^n}=\\ &=2\int_0^{\pi}\frac{dx}{(\frac{a^2+b^2}{2}+\frac{a^2-b^2}{2}\cos 2x)^n}=\\ &=\frac{1}{i}\oint_{|z|=1}\frac{z^{-1}dz}{(\frac{a^2+b^2}{2}+\frac{a^2-b^2}{2}\frac{z+z^{-1}}{2})^n}=\\ &=\frac{2^{2n}}{(a^2-b^2)^n}\frac{1}{i}\oint_{|z|=1}\frac{z^{-1}dz}{(z+z^{-1}-c-c^{-1})^n}=\\ &=\frac{2^{2n}}{(a^2-b^2)^n}\frac{1}{i}\oint_{|z|=1}\frac{z^{n-1}dz}{(z-c)^n(z-c^{-1})^n} \end{align} where the parameter $c$ is defined by the equation $$c+c^{-1}=2\frac{b^2+a^2}{b^2-a^2}.$$ In particular, we can (and will) set $\displaystyle c=\frac{b-a}{b+a}$. Now for $b>a>0$ the point $z=c$ is inside, and $z=c^{-1}$ outside the unit circle. Therefore, by residues, \begin{align} I_n&=2\pi\cdot\frac{2^{2n}}{(a^2-b^2)^n}\cdot\mathrm{res}_{z=c}\frac{z^{n-1}}{(z-c)^n(z-c^{-1})^n}=\\ &=\frac{2\pi}{(n-1)!}\cdot\frac{2^{2n}}{(a^2-b^2)^n}\cdot\left[\frac{d^{n-1}}{dz^{n-1}} \frac{z^{n-1}}{(z-\frac{b+a}{b-a})^n}\right]_{z=\frac{b-a}{b+a}} \end{align} In particular, \begin{align} I_1&=\frac{2\pi}{ab},\\ I_2&=\frac{\pi(a^2+b^2)}{a^3b^3},\\ I_3&=\frac{\pi(3a^4+2a^2b^2+3b^4)}{4a^5b^5},\\ &\ldots \end{align}

Explicit computation (by RGB)

We can compute the residue explicitly by using the binomial formula.

We have that $$z^{n-1}=c^{n-1}\left(1+\frac{z-c}{c}\right)^{n-1}=\sum_{k=0}^{\infty}\binom{n-1}{k}\frac{(z-c)^k}{c^{k-n+1}},$$

and that $$(z-c^{-1})^{-n}=(c-c^{-1})^{-n}\left(1+\frac{z-c}{c-c^{-1}}\right)^{-n}=\sum_{k=0}^{\infty}\binom{-n}{k}\frac{(z-c)^k}{\left(c-c^{-1}\right)^{k+n}}.$$

We multiply these two to get a series $\sum_{k=0}^{\infty}A_k(z-c)^k$. We need then to divide by $(z-c)^n$ and get the coefficient of the term with degree $-1$. This will be the coefficient $A_{n-1}$.

Using the formula for the product of two series we get

$$A_{n-1}=\sum_{r=0}^{n-1}\frac{\binom{n-1}{r}}{c^{r-n+1}}\frac{\binom{-n}{n-1-r}}{\left(c-c^{-1}\right)^{2n-r-1}}.$$

Finally we get $$I_n=2\pi\cdot\frac{2^{2n}}{(a^2-b^2)^n}\cdot\sum_{r=0}^{n-1}\frac{\binom{n-1}{r}}{c^{r-n+1}}\frac{\binom{-n}{n-1-r}}{\left(c-c^{-1}\right)^{2n-r-1}}.$$

Explicit computation II (by O.L.)

Let us write $$\frac{z^{n-1}}{(z-c^{-1})^n}=\frac{((z-c^{-1})+c^{-1})^{n-1}}{(z-c^{-1})^n}=\sum_{k=0}^{n-1}{n-1\choose k}c^{k+1-n}(z-c^{-1})^{k-n}$$ We have to compute $(n-1)$th derivative of this expression and then evaluate it at $z=c$. This gives the formula $$I_n=2\pi\cdot\frac{2^{2n}}{(a^2-b^2)^n}\cdot\sum_{k=0}^{n-1}(-1)^{n-1}{n-1\choose k}{2n-2-k\choose n-1}c^{k+1-n}(c-c^{-1})^{k-2n+1}$$ which is equivalent to that of RGB.

Here is a different solution using differentiation under integral sign. We firstly look at the the case $n=1$, then \begin{align*} I_1(\alpha,\beta) & = \int_{0}^{2\pi}\frac{dx}{\alpha \cos^2x+\beta \sin^2x} \\ & = \int_{0}^{2\pi} \frac{\sec^2x}{\alpha + \beta \tan^2x} \,\mathrm{d}x\\ & = \frac{1}{\beta} \int_0^{2\pi} \frac{1}{\left(\sqrt{\frac{\alpha}{\beta}}\right)^2 + \tan^2 (x)}\; \mathrm{d}(\tan x)\,\\ & = \frac{1}{\sqrt{\alpha \beta}} \left[ \left(\tan^{-1}\left(\sqrt{\frac{\beta}{\alpha}}\tan (x)\right)\right) \right]_0^{2\pi} = \frac{2\pi}{\sqrt{\alpha\beta}} \end{align*} Where $a^2 = \alpha$ and $b^2 = \beta$. Differentiating $I_n$ with respect to $\alpha$ and $\beta$ yields \begin{align*} \frac{\mathrm{d}}{\mathrm{d}\alpha} I_n & = - \int_0^{2\pi} \frac{n \cos^2x}{(\alpha \cos^2x + \beta \sin^2x)^{n+1}} \\ \frac{\mathrm{d}}{\mathrm{d}\beta} I_n & = - \int_0^{2\pi} \frac{n \sin^2x}{(\alpha \cos^2x + \beta \sin^2x)^{n+1}} \end{align*} Addition gives that \begin{align} \nabla I_n = \left( \frac{\mathrm{d}}{\mathrm{d}\alpha} + \frac{\mathrm{d}}{\mathrm{d}\beta} \right) I_1 = -n \int_0^{2\pi} \frac{(\sin^2x + \cos^2x)\mathrm{d}x}{(\alpha \cos^2x + \beta \sin^2x)^2} = -n I_{n+1} \end{align} Where $\nabla$ is the nabla-operator. This gives us the recurrence relation \begin{align} \nabla I_{n-1} = (1-n) I_n \tag{1} \end{align} Which can be used to evaluate each $n \in \mathbb{N} / \{ 1 \}$. This is a fine answer, but if one wants an explicit expression it can be shown by induction that $$ \boxed{ \displaystyle I_{n+1} = \frac{2\pi(-1)^{n}}{n!} \left( \frac{\partial }{\partial \alpha} + \frac{\partial}{\partial \beta} \right)^n \frac{1}{\sqrt{\alpha\beta}} } $$ Where the last part is the binomial expansion of the partial derivatives, and this holds for all $n \in \mathbb{N}$. Example for $n=3$ we have $$ \left( \frac{\partial }{\partial \alpha} + \frac{\partial}{\partial \beta} \right)^2 f(\alpha,\beta) = \left(\frac{\partial^2}{\partial \alpha^2}+2\frac{\partial^2}{\partial \alpha \, \partial \beta}+\frac{\partial^2}{\partial \beta^2}\right)f(\alpha,\beta) $$ We can also express the partial derivatives as a sum like $$ \left( \frac{\partial }{\partial \alpha} + \frac{\partial}{\partial \beta} \right)^n = \sum_{k=0}^b \binom{n}{k} \frac{\partial^{n-k} }{\partial \alpha^{n-k} } \frac{\partial^{k} }{\partial \beta^{k} } = \sum_{k=0}^\infty \binom{n}{k} \frac{\partial^{n} }{\partial \alpha^{n-k} \,\partial^k \beta} $$ without that doing us much good. For actually computing explicit values of $n$ the recurrence relation is more convenient.

EDIT: To give an intuition of the "closed" expression one can first rewrite $(1)$ to $$ I_n = - \frac{1}{n-1} \nabla I_{n-1} $$ Direct computation of the first values gives One can also show that the expression satisfies $(1)$ directly by insertion, and the use of the chain rule.