How to evaluate $\int_0^{2\pi} \frac{d\theta}{A+B\cos\theta}$?

To avoid confusion with limits, note that $$\int_0^{\pi} \dfrac{dx}{a+b \cos(x)} = \int_{\pi}^{2\pi} \dfrac{dx}{a+b \cos(x)}$$ Hence, we have $$I = \int_0^{2\pi} \dfrac{dx}{a+b \cos(x)} = 2\int_0^{\pi} \dfrac{dx}{a+b \cos(x)}$$ Now use your substitution $t = \tan(x/2)$ and note that $t$ goes from $0$ to $\infty$ as $x$ goes from $0$ to $\pi$.

EDIT

An easier way if you are familiar with a little bit of complex analysis is as follows.

Let $z = e^{ix}$. Then we have that $dz = iz dx$. Hence, \begin{align} I & = \int_0^{2 \pi} \dfrac{dx}{a+b \cos(x)} = \oint_{\vert z \vert = 1} \dfrac{dz}{iz \left( a + \dfrac{b}2 \left(z+ \dfrac1z \right)\right)}\\ & = \dfrac1i \oint_{\vert z \vert = 1} \dfrac{dz}{az + \dfrac{b}2 \left(z^2+1 \right)} = \dfrac2{ib} \oint_{\vert z \vert = 1}\dfrac{dz}{z^2 + \dfrac{2a}b z + 1}\\ & = \dfrac2{ib} \oint_{\vert z \vert = 1} \dfrac{dz}{\left(z + \dfrac{a}b \right)^2 + 1 - \dfrac{a^2}{b^2}} = \dfrac2{ib} \oint_{\vert z \vert = 1} \dfrac{dz}{\left(z + r + \sqrt{r^2-1} \right)\left(z + r - \sqrt{r^2-1} \right)} \end{align} where $r = \dfrac{a}b \in \mathbb{R}$.

First note that if $\vert r \vert \leq 1$, then $\left \vert r \pm \sqrt{r^2-1} \right \vert = 1$. Hence, the integral doesn't exist.

If $r > 1$, then $ \left \vert r + \sqrt{r^2 - 1} \right \vert > 1$ and $\left \vert r - \sqrt{r^2-1} \right \vert < 1$. Hence, by Cauchy integral formula, we have that $$I = \dfrac{2}{ib} 2 \pi i \dfrac1{2 \sqrt{r^2-1}} = \dfrac{2 \pi \, \text{sign}(a)}{\sqrt{a^2 - b^2}}$$

If $r < -1$, then $\left \vert r - \sqrt{r^2 - 1} \right \vert > 1$ and $\left \vert r + \sqrt{r^2-1} \right \vert < 1$. Hence, by Cauchy integral formula, we have that $$I = \dfrac{2}{ib} 2 \pi i \dfrac{-1}{2 \sqrt{r^2-1}} = \dfrac{2 \pi \, \text{sign}(a)}{\sqrt{a^2 - b^2}}$$ Hence, to summarize, $$I = \begin{cases} \dfrac{2 \pi \, \text{sign}(a)}{\sqrt{a^2 - b^2}} & \text{if } \vert a \vert > \vert b\\ \text{Does not exists} & \text{if } \vert a \vert \leq \vert b\end{cases}$$

Ideally, you want to do your substitution on a domain where it is defined, continuous, and invertible.

That's not quite true here, as this plot shows.

But we can carefully break the domain up into parts where the substitution behaves nicely, sort of, by splitting $\theta \in [0, 2 \pi]$ into two regions $\theta \in [0, \pi]$ and $\theta \in [\pi, 2 \pi]$. Note that we really should break the integral up into two integrals, one over each domain.

Over $\theta \in [0,\pi]$, $t$ varies from $0$ to $+\infty$; so the substitution is problematic! However, you can still set up the improper integral, and take care to show that it converges to the right thing.

You can treat $\theta \in [\pi, 2\pi]$ similarly.

Of course, once you've set that up, it's clear you can put the integrals back together, since $\int_0^{+\infty} f + \int_{-\infty}^0 f = \int_{-\infty}^{+\infty} f$. You could have observed this by changing the original limits of integration from $[0, 2\pi]$ to $[-\pi, \pi]$ because the integrand was periodic.

P.S. there's one extra problem with your integral: the integrand diverges at the $\theta$ where $A + B \cos \theta = 0$. Such solutions exist if $|A| \leq |B|$. So you have to handle the improperness of that as well. (notice that your solution method for the transformed integral assumes this, and possibly more)