2 subtle problems on calculating this integral

$\newcommand{\arcsec}{\operatorname{arcsec}}$ Well, arcsecant is nasty, and I personally try to avoid it at any cost!

Now, the same but on a more serious note. There are two competing definitions of the arcsecanrt function. When you use this function (and I still suggest that you don't!), then you have to be clear as to which of the two conventions you're using, and then you have to stay consistent with that convention.

• One option is to define the range of $\arcsec$ as $\color{blue}{\left[0,\frac{\pi}{2}\right)\cup\left(\frac{\pi}{2},\pi\right]}$. The advantage is that it's consistent with the formula $\arcsec(x)=\arccos(1/x)$. But in this case, that radical simplifies differently, with an absolute value: $\sqrt{\sec(t)-1}=|\tan(t)|$, to make sure that it's true for all $t\in\left[0,\frac{\pi}{2}\right)\cup\left(\frac{\pi}{2},\pi\right]$.

• Another option is to define the range of $\arcsec$ as $\color{blue}{\left[0,\frac{\pi}{2}\right)\cup\left[\pi,\frac{3\pi}{2}\right)}$. The advantage is that now it's easier to simplify that radical: $\sqrt{\sec(t)-1}=\tan(t)$ for all $t\in\left[0,\frac{\pi}{2}\right)\cup\left[\pi,\frac{3\pi}{2}\right)$. But we don't have the relation ``$\arcsec(x)=\arccos(1/x)$'' anymore, because of the different ranges of these two functions in this case.

If you use the first convention, then simplifying the square root produces that absolute value. If you use the second convention, then you made a mistake when converting from $\arcsec$ to $\arccos$. Fixing that mistake will introduce the absolute value at this point.

In the second case of your attempt in the Edit, you have $t\in(\frac\pi2, \pi]$ and $u = a\sec t \in (-\infty, -a]$. Then, we have $t-\frac\pi2 \in [0,\frac\pi2)$ and

$$\sin(t-\frac{\pi}2) = -\cos t =-\frac1{\sec t} =-\frac a{u}=\frac a{|u|}$$

which leads to

$$ t -\frac\pi2= \sin^{-1}\frac a{|u|}$$

Now, apply the identity $\sin^{-1} x + \cos^{-1}x = \frac\pi2$ to express $t$ as

$$t = \pi - \cos^{-1}\frac 1{|u|}$$

Then, plug the result into the integral for $u \in (-\infty, -a]$

$$\int\frac{\sqrt{u^2-a^2}}{u}du= - a\tan t + at = \sqrt{u^2-a^2} -a\cos^{-1}\frac a{|u|}+ a\pi+C$$

where the constant term $a\pi$ is to be absorbed in $C$. Moreover, note that

$$\arccos(\frac{a}{u}) \ne -\arccos(\frac{a}{-u})$$

Instead, they differ by the constant $\pi$,

$$\arccos(\frac{a}{u}) = \pi - \arccos(\frac{a}{-u})$$