Why doesn't trig substitution work for definite integrals?

The part of the statement of the Fundamental theorem of calculus on which the proof of the substitution theorem relies has no requirement that the change of variable be a bijection. This is a common error. (I made it myself until I first taught Calculus.) (I'll return to this point at the end of answering your Question because this Question gives the appearance that bijectivity is required.)

Another common error is to believe that, during trigonometric substitution of a definite integral, one "pushes" the bounds of integration from the old integral to the new integral along the change of variable. This is true when one uses the substitution theorem in the forward direction and false when used in reverse, as it is in trigonometric substitution. One must "pull" the bounds along the inverse function of the substitution (with whatever domain restriction it has) to the new bounds. In your case, $x = r \sin \theta$ gives the inverse $\theta = \arcsin(x/r)$. So you have no choice but to find the $\theta$s \begin{align*} \theta_{\text{lower}} &= \arcsin(-1) = -\pi/2 \\ \theta_{\text{upper}} &= \arcsin(1) = \pi/2 \text{.} \end{align*} You could use a different restriction of the domain of sine for your arcsine, but to apply the Fundamental Theorem of Calculus, this choice of inverse function must be continuous (hence defined) on the interval of integration. There is no such restriction giving an arcsine that contains the interval $[-5\pi/2, 5\pi/2]$. This is where bijectivity enters the picture -- to ensure the inverse function exists over the entire interval of integration. (And it only enters the picture when applying the theorem in reverse.)

We need to fix the exceedingly common algebra error in your derivation. In particular, $\sqrt{a^2} = |a|$ for any $a$. Let's take that square root correctly. \begin{align*} I &= \int_{-r}^r \; \sqrt{r^2 - x^2} \,\mathrm{d}x & & \hspace{-1in}\begin{bmatrix} x = r \sin \theta \\ \mathrm{d}x = r \cos \theta \,\mathrm{d}\theta \\ \theta = \sin^{-1}(x/r) \end{bmatrix} \\ &= \int_{\sin^{-1}(-r/r)}^{\sin^{-1}(r/r)} \; \sqrt{r^2 - (r \sin \theta)^2}\, r \cos \theta \,\mathrm{d}\theta \\ &= \int_{-\pi/2}^{\pi/2} \; \sqrt{r^2(1 - \sin^2 \theta)}\, r \cos \theta \,\mathrm{d}\theta \\ &= \int_{-\pi/2}^{\pi/2} \; \sqrt{r^2 \cos^2 \theta}\, r \cos \theta \,\mathrm{d}\theta \\ &= \int_{-\pi/2}^{\pi/2} \; |r \cos \theta| r \cos \theta \,\mathrm{d}\theta \\ &= \int_{-\pi/2}^{\pi/2} \; |r| |\cos \theta| r \cos \theta \,\mathrm{d}\theta \\ &= \int_{-\pi/2}^{\pi/2} \; |\cos \theta| r^2 \cos \theta \,\mathrm{d}\theta & & \hspace{-1in}[ r \geq 0 ] \\ \end{align*} Recall that cosine is nonnegative on our interval of integration, quadrants I and IV and the quadrantal angle $0$. \begin{align*} I &= r^2 \int_{-\pi/2}^{\pi/2} \; \cos^2 \theta \,\mathrm{d}\theta \\ &= r^2 \left.\left( \frac{\theta}{2} + \frac{\sin(2\theta)}{4} \right) \right|_{\theta = -\pi/2}^{\pi/2} \\ &= r^2 \left( \left( \frac{\pi}{4} + \frac{\sin(\pi)}{4} \right) - \left( \frac{-\pi}{4} + \frac{\sin(-\pi)}{4} \right) \right) \\ &= r^2 \left( \frac{\pi}{4} + \frac{\pi}{4} \right) \\ &= \frac{1}{2} \pi r^2 \text{.} \end{align*}

An actual statement of the substitution theorem (footnote) is this:

Let $\varphi:[a,b] \rightarrow I$ be a differentiable function with a continuous derivative, where $I \subseteq \Bbb{R}$ is an interval. Suppose that $f:I \rightarrow \Bbb{R}$ is a continuous function. Then, if $u = \varphi(x)$ $$ \int_a^b \; f \left( \varphi(x) \right) \varphi'(x) \,\mathrm{d}x = \int_{\varphi(a)}^{\varphi(b)} \; f(u) \,\mathrm{d}u \text{.} $$

That English Wikipedia article also explains why trigonometric substitution is a little different from normal substitution.

The formula is used to transform one integral into another integral that is easier to compute. Thus, the formula can be read from left to right or from right to left in order to simplify a given integral. When used in the former manner, it is sometimes known as $u$-substitution or $w$-substitution in which a new variable is defined to be a function of the original variable found inside the composite function multiplied by the derivative of the inner function. The latter manner is commonly used in trigonometric substitution, replacing the original variable with a trigonometric function of a new variable and the original differential with the differential of the trigonometric function.

Let me give two examples to really clarify what is meant by right-to-left and left-to-right. First, left-to-right: $$ J = \int_{-2}^{1} \frac{2 x \, \mathrm{d}x}{\sqrt{x^2 + 1}} \text{.} $$ Here, we choose $u$ to stand for some combination of the dummy variable, making parts of the integrand simpler and we hope that the resulting differential will consume part of the integrand, leaving something simpler. So we set $u = x^2 + 1$. That is, $\varphi(x) = x^2 + 1$. This is not a bijection. $\varphi(1) = \varphi(-1)$. Then, $\mathrm{d}u = 2 x \,\mathrm{d}x$. That is, $\varphi'(x) = 2x$. We are matching to the left side of the substitution equation. So we apply the theorem and obtain the version on the right-hand side of the equation: $$ J = \int_{\varphi(-2) = 5}^{\varphi(1) = 2} \frac{\mathrm{d}u}{\sqrt{u}} \text{,} $$ which is easy to finish. Now, right-to-left: $$ K = \int_{-r}^r \; \sqrt{r^2 - x^2} \,\mathrm{d}x \text{.} $$ Here, we set $x = r \sin \theta$, making no attempt to capture some combination of the dummy variable and making no attempt to capture a piece of the integrand by means of the differential. We start with the simpler looking integral on the right of the equation in the theorem and replace it with the elaborated integral on the left. This is why the inverse function has to work on the entire interval, because we have to be able to carry the endpoints of integration along the inverse function, to the endpoints of the interval $I$ (notation from the theorem).

When using the theorem right-to-left, one applies $\varphi^{-1}$ to the endpoints. In your Question, there is no choice of $\varphi^{-1}$ that gives an interval $I$ with the endpoints $-5\pi/2$ and $5\pi/2$. The maximal interval for any inverse of sine has width $\pi$.

(footnote) One might be (justifiably) concerned that the statement of the theorem at the English Wikipedia is incomplete.

Rogowski et al. "Calculus: Early Transcendentals, 4th ed.", p. 340:

Theorem 1: The Substitution Method: If $F'(x) = f(x)$, and $u$ is a differentiable function whose range includes the domain of $f$, $$ \int f(u(x))u'(x)\,\mathrm{d}x = F(u(x)) + C \text{.} $$

(Since $F$ is an antiderivative of $f$, the object on the right is $\int f(u) \,\mathrm{d}u$.)

No bijectivity requirement here. You will also not find a bijectivity requirement in other careful statements of the theorem because there is no explicit or implicit bijectivity requirement in the forward direction. Bijectivity is only relevant in the reverse direction and is expressed implicitly by means of the interval $I$, in the English Wikipedia quote, and in Rogowski by writing $F(u(x))$, which requires the domain of $F$ to include the range of $u$, which is given to include the domain of $f$. (Recall that in a definite integral, we may replace "$f$" with a function that is undefined outside the interval of integration.)


Ideally, as others have pointed out, you will set up the bounds of your trigonometric substitution so that each value of $x$ you need is produced once and only once, in a continuous stream. For $-\frac\pi2 \leq \theta \leq \frac\pi2,$ if $x = r\sin\theta$ then $x$ increases monotonically from $-r$ to $r$ as $\theta$ increases monotonically from $-\frac\pi2$ to $\frac\pi2.$ This makes a very "clean" substitution.

(Note: throughout this answer I am assuming $r > 0.$ In the case where $r<0$ then $\int_{-r}^r\sqrt{r^2-x^2}dx$ is negative and is not the answer to the initial problem, "What is the area of a semicircle?")

When you increase $\theta$ from $-\frac{5\pi}2$ to $\frac{5\pi}2,$ the value of $x$ starts at $-r,$ increases to $r,$ then decreases back to $-r$, then increases to $r$ again, decreases to $-r$ again, and finally increases to $r.$ That's a lot of increasing and decreasing just to cover the distance from $-r$ to $r.$

But the up-and-down-and-up movement of $x$ is not really the problem. What is the problem is that your substitution is not correct over the entire domain $-\frac{5\pi}2 \leq \theta \leq \frac{5\pi}2.$

In particular, look at this equation on which you rely (where I have written $a$ and $b$ as the bounds of the interval of integration, since you propose to use the same method from $-\frac{5\pi}2$ to $\frac{5\pi}2$ as for $-\frac{\pi}2$ to $\frac{\pi}2$):

$$\int_a^b r\cos\theta \sqrt{r^2-r^2\sin^2\theta} \,d\theta =\int_a^b r^2\cos^2\theta\,d\theta.$$

In order to justify this equation, you must show that $\sqrt{r^2-r^2\sin^2\theta} = r\cos\theta.$ That is easily proved when $\cos\theta \geq 0,$ but it is false when $\cos\theta < 0.$ When $\cos\theta < 0,$ the correct equation is

$$\int_a^b r\cos\theta \sqrt{r^2-r^2\sin^2\theta} \,d\theta =\int_a^b -r^2\cos^2\theta\,d\theta.$$

Alternatively, you could combine the two equations as $$\int_a^b r\cos\theta \sqrt{r^2-r^2\sin^2\theta} \,d\theta =\int_a^b -r^2\cos\theta \lvert\cos\theta\rvert \,d\theta,$$ but the integral of $\cos\theta \lvert\cos\theta\rvert$ is not the same as the integral of $\cos^2\theta,$ so you still have some work to do to sort things out.

Here's how the integral can be correctly integrated from $-\frac{5\pi}2$ to $\frac{5\pi}2$:

\begin{align} \int_{-r}^r \sqrt{r^2-x^2}\,dx &= \int_{-5\pi/2}^{5\pi/2} r\cos\theta\sqrt{r^2-r^2\sin^2\theta} \,d\theta\\ &= \int_{-5\pi/2}^{5\pi/2} r^2(\cos\theta)\lvert\cos\theta\rvert \,d\theta\\ &= \int_{-5\pi/2}^{-3\pi/2} r^2\cos^2\theta \,d\theta\\ &\qquad + \int_{-3\pi/2}^{-\pi/2} -r^2\cos^2\theta \,d\theta\\ &\qquad + \int_{-\pi/2}^{\pi/2} r^2\cos^2\theta \,d\theta\\ &\qquad + \int_{\pi/2}^{3\pi/2} -r^2\cos^2\theta \,d\theta\\ &\qquad + \int_{3\pi/2}^{5\pi/2} r^2\cos^2\theta \,d\theta\\ &= \frac{\pi r^2}{2} - \frac{\pi r^2}{2} + \frac{\pi r^2}{2} - \frac{\pi r^2}{2} + \frac{\pi r^2}{2} \\ &= \frac{\pi r^2}{2}, \end{align} using the fact that $\lvert\cos\theta\rvert = -\cos\theta$ when $\cos\theta \leq 0.$

You actually get the correct answer, but only if you integrate the correct function over the entire interval. Also notice that each time $\sin\theta$ decreases from $1$ to $-1$ (that is, each time $x$ decreases from $r$ to $-r$) you precisely wipe out the amount you integrated on the previous increase. In effect, by allowing $x$ to go up and down all these times, you end up integrating

\begin{multline} \int_{-r}^r \sqrt{r^2-x^2}\,dx + \int_r^{-r} \sqrt{r^2-x^2}\,dx + \int_{-r}^r \sqrt{r^2-x^2}\,dx\\ + \int_r^{-r} \sqrt{r^2-x^2}\,dx + \int_{-r}^r \sqrt{r^2-x^2}\,dx, \end{multline}

in which the first four integrals cancel each other out.