On the integral $\int_0^1\frac{\arctan\sqrt{t^2+a}}{(t^2+b)\sqrt{t^2+a}}dt$

Solution 1:

Define the function $\mathcal{I}:\mathbb{R}_{>0}^{2}\rightarrow\mathbb{R}$ via the definite integral

$$\mathcal{I}{\left(p,q\right)}:=\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\sqrt{t^{2}+p^{2}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}.\tag{1}$$

We seek a closed-form expression for $\mathcal{I}{\left(p,q\right)}$ for $\left(p,q\right)\in\mathbb{R}_{>0}^{2}$ such that $p>q$. It should be clear this problem is completely equivalent to the one posed by the OP.

We will make use of the following Euler substitution:

$$\sqrt{t^{2}+p^{2}}=t+x;~~~\small{p>0\land t\in\mathbb{R}\land x\in\mathbb{R}_{>0}}.\tag{2}$$

Solving for $t$, we have

$$t=\frac{p^{2}-x^{2}}{2x},$$

$$\implies\sqrt{t^{2}+p^{2}}=t+x=\frac{p^{2}+x^{2}}{2x},$$

$$\implies dt=dx\,\frac{(-1)\left(p^{2}+x^{2}\right)}{2x^{2}}.$$

Suppose $\left(p,q\right)\in\mathbb{R}_{>0}^{2}\land p>q$. We find

$$\begin{align} \mathcal{I}{\left(p,q\right)} &=\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\sqrt{t^{2}+p^{2}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\int_{0}^{1}\mathrm{d}t\,\frac{\left[\frac{\pi}{2}-\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}\right]}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi}{2}\int_{0}^{1}\mathrm{d}t\,\frac{1}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi}{2}\int_{0}^{\frac{1}{\sqrt{1+p^{2}}}}\mathrm{d}u\,\frac{1}{\left(1-u^{2}\right)}\cdot\frac{1}{\left(\frac{pu}{\sqrt{1-u^{2}}}\right)^{2}+q^{2}};~~~\small{\left[t=\frac{pu}{\sqrt{1-u^{2}}}\right]}\\ &~~~~~-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi}{2}\int_{0}^{\frac{1}{\sqrt{1+p^{2}}}}\mathrm{d}u\,\frac{1}{p^{2}u^{2}+q^{2}\left(1-u^{2}\right)}-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi}{2}\int_{0}^{\frac{1}{\sqrt{1+p^{2}}}}\mathrm{d}u\,\frac{1}{q^{2}+\left(p^{2}-q^{2}\right)u^{2}}-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi}{2}\int_{0}^{\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}}\mathrm{d}v\,\frac{q}{\sqrt{p^{2}-q^{2}}}\cdot\frac{1}{q^{2}+q^{2}v^{2}};~~~\small{\left[u=\frac{qv}{\sqrt{p^{2}-q^{2}}}\right]}\\ &~~~~~-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi}{2}\cdot\frac{1}{q\sqrt{p^{2}-q^{2}}}\int_{0}^{\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}}\mathrm{d}v\,\frac{1}{1+v^{2}}-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}.\tag{3a}\\ \end{align}$$

Then $0<-1+\sqrt{1+p^{2}}<p$, and

$$\begin{align} \mathcal{I}{\left(p,q\right)} &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}\\ &~~~~~-\int_{p}^{-1+\sqrt{1+p^{2}}}\mathrm{d}x\,\frac{(-1)\left(p^{2}+x^{2}\right)}{2x^{2}}\cdot\frac{1}{\left(\frac{p^{2}-x^{2}}{2x}\right)^{2}+q^{2}}\cdot\frac{2x}{p^{2}+x^{2}}\\ &~~~~~\times\arctan{\left(\frac{2x}{p^{2}+x^{2}}\right)};~~~\small{\left[-t+\sqrt{t^{2}+p^{2}}=x\right]}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\int_{-1+\sqrt{1+p^{2}}}^{p}\mathrm{d}x\,\frac{4x\arctan{\left(\frac{2x}{p^{2}+x^{2}}\right)}}{\left(p^{2}-x^{2}\right)^{2}+4q^{2}x^{2}}.\tag{3b}\\ \end{align}$$

It follows from the arctangent addition formula that

$$\arctan{\left(\frac{x}{-1+\sqrt{1+p^{2}}}\right)}-\arctan{\left(\frac{x}{1+\sqrt{1+p^{2}}}\right)}=\arctan{\left(\frac{2x}{p^{2}+x^{2}}\right)};~~~\small{p\in\mathbb{R}_{>0}\land x\in\mathbb{R}}.\tag{4}$$

Set $r:=\frac{p}{1+\sqrt{1+p^{2}}}\land\sigma:=\arcsin{\left(\frac{q}{p}\right)}$. Then, $0<r<1\land0<\sigma<\frac{\pi}{2}\land r^{-1}=\frac{1+\sqrt{1+p^{2}}}{p}=\frac{p}{-1+\sqrt{1+p^{2}}}$, and

$$\begin{align} \mathcal{I}{\left(p,q\right)} &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\int_{-1+\sqrt{1+p^{2}}}^{p}\mathrm{d}x\,\frac{4x\arctan{\left(\frac{2x}{p^{2}+x^{2}}\right)}}{\left(p^{2}-x^{2}\right)^{2}+4q^{2}x^{2}}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\int_{-1+\sqrt{1+p^{2}}}^{p}\mathrm{d}x\,\frac{4x}{\left(p^{2}-x^{2}\right)^{2}+4q^{2}x^{2}}\\ &~~~~~\times\left[\arctan{\left(\frac{x}{-1+\sqrt{1+p^{2}}}\right)}-\arctan{\left(\frac{x}{1+\sqrt{1+p^{2}}}\right)}\right]\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\int_{\frac{-1+\sqrt{1+p^{2}}}{p}}^{1}\mathrm{d}y\,\frac{4p^{2}y}{\left(p^{2}-p^{2}y^{2}\right)^{2}+4p^{2}q^{2}y^{2}}\\ &~~~~~\times\left[\arctan{\left(\frac{py}{-1+\sqrt{1+p^{2}}}\right)}-\arctan{\left(\frac{py}{1+\sqrt{1+p^{2}}}\right)}\right];~~~\small{\left[x=py\right]}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\frac{1}{p^{2}}\int_{\frac{-1+\sqrt{1+p^{2}}}{p}}^{1}\mathrm{d}y\,\frac{4y}{\left(1-y^{2}\right)^{2}+4\left(\frac{q}{p}\right)^{2}y^{2}}\\ &~~~~~\times\left[\arctan{\left(\frac{py}{-1+\sqrt{1+p^{2}}}\right)}-\arctan{\left(\frac{py}{1+\sqrt{1+p^{2}}}\right)}\right]\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\frac{1}{p^{2}}\int_{r}^{1}\mathrm{d}y\,\frac{4y}{\left(1-y^{2}\right)^{2}+4y^{2}\sin^{2}{\left(\sigma\right)}}\\ &~~~~~\times\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\frac{1}{p^{2}}\int_{r}^{1}\mathrm{d}y\,\frac{4y\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]}{1-2y^{2}\left[1-2\sin^{2}{\left(\sigma\right)}\right]+y^{4}}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\frac{1}{p^{2}}\int_{r}^{1}\mathrm{d}y\,\frac{4y\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]}{1-2y^{2}\cos{\left(2\sigma\right)}+y^{4}}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}\\ &~~~~~-\frac{1}{p^{2}}\int_{r}^{1}\mathrm{d}y\,\frac{4y\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]}{\left[1-2y\cos{\left(\sigma\right)}+y^{2}\right]\left[1+2y\cos{\left(\sigma\right)}+y^{2}\right]}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}\\ &~~~~~-\frac{1}{p^{2}\cos{\left(\sigma\right)}}\int_{r}^{1}\mathrm{d}y\,\frac{4y\cos{\left(\sigma\right)}}{\left[1-2y\cos{\left(\sigma\right)}+y^{2}\right]\left[1+2y\cos{\left(\sigma\right)}+y^{2}\right]}\\ &~~~~~\times\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}\\ &~~~~~-\frac{1}{p^{2}\cos{\left(\sigma\right)}}\int_{r}^{1}\mathrm{d}y\,\left[\frac{1}{1-2y\cos{\left(\sigma\right)}+y^{2}}-\frac{1}{1+2y\cos{\left(\sigma\right)}+y^{2}}\right]\\ &~~~~~\times\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}\\ &~~~~~-\frac{1}{p^{2}\sin{\left(\sigma\right)}\cos{\left(\sigma\right)}}\int_{r}^{1}\mathrm{d}y\,\left[\frac{\sin{\left(\sigma\right)}}{1-2y\cos{\left(\sigma\right)}+y^{2}}-\frac{\sin{\left(\sigma\right)}}{1+2y\cos{\left(\sigma\right)}+y^{2}}\right]\\ &~~~~~\times\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}\\ &~~~~~-\frac{1}{q\sqrt{p^{2}-q^{2}}}\int_{r}^{1}\mathrm{d}y\,\left[\frac{\sin{\left(\sigma\right)}}{1-2y\cos{\left(\sigma\right)}+y^{2}}-\frac{\sin{\left(\sigma\right)}}{1+2y\cos{\left(\sigma\right)}+y^{2}}\right]\\ &~~~~~\times\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right].\tag{5}\\ \end{align}$$

As such, let's introduce another auxiliary function $\mathcal{J}:\left(0,1\right)\times\left(0,\frac{\pi}{2}\right)\rightarrow\mathbb{R}$ defined via the last definite integral above:

$$\mathcal{J}{\left(r,\sigma\right)}:=\int_{r}^{1}\mathrm{d}y\,\left[\frac{\sin{\left(\sigma\right)}}{1-2y\cos{\left(\sigma\right)}+y^{2}}-\frac{\sin{\left(\sigma\right)}}{1+2y\cos{\left(\sigma\right)}+y^{2}}\right]\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right].\tag{6}$$

Given $\left(r,\sigma\right)\in\left(0,1\right)\times\left(0,\frac{\pi}{2}\right)$, we find

$$\begin{align} \mathcal{J}{\left(r,\sigma\right)} &=\int_{r}^{1}\mathrm{d}y\,\left[\frac{\sin{\left(\sigma\right)}}{1-2y\cos{\left(\sigma\right)}+y^{2}}-\frac{\sin{\left(\sigma\right)}}{1+2y\cos{\left(\sigma\right)}+y^{2}}\right]\\ &~~~~~\times\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]\\ &=\int_{r}^{1}\mathrm{d}y\,\left[\frac{\sin{\left(\sigma\right)}}{1-2y\cos{\left(\sigma\right)}+y^{2}}-\frac{\sin{\left(\sigma\right)}}{1+2y\cos{\left(\sigma\right)}+y^{2}}\right]\arctan{\left(\frac{y}{r}\right)}\\ &~~~~~-\int_{r}^{1}\mathrm{d}y\,\left[\frac{\sin{\left(\sigma\right)}}{1-2y\cos{\left(\sigma\right)}+y^{2}}-\frac{\sin{\left(\sigma\right)}}{1+2y\cos{\left(\sigma\right)}+y^{2}}\right]\arctan{\left(ry\right)}\\ &=\int_{1}^{\frac{1}{r}}\mathrm{d}t\,\left[\frac{r\sin{\left(\sigma\right)}}{1-2rt\cos{\left(\sigma\right)}+r^{2}t^{2}}-\frac{r\sin{\left(\sigma\right)}}{1+2rt\cos{\left(\sigma\right)}+r^{2}t^{2}}\right]\arctan{\left(t\right)};~~~\small{\left[y=rt\right]}\\ &~~~~~-\int_{r^{2}}^{r}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\arctan{\left(u\right)};~~~\small{\left[y=\frac{u}{r}\right]}\\ &=\int_{r}^{1}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\\ &~~~~~\times\arctan{\left(\frac{1}{u}\right)};~~~\small{\left[t=u^{-1}\right]}\\ &~~~~~-\int_{r^{2}}^{r}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\arctan{\left(u\right)}\\ &=\int_{r}^{1}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\left[\frac{\pi}{2}-\arctan{\left(u\right)}\right]\\ &~~~~~-\int_{r^{2}}^{r}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\arctan{\left(u\right)}\\ &=\frac{\pi}{2}\int_{r}^{1}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\\ &~~~~~-\int_{r}^{1}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\arctan{\left(u\right)}\\ &~~~~~-\int_{r^{2}}^{r}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\arctan{\left(u\right)}\\ &=\frac{\pi}{2}\int_{r}^{1}\mathrm{d}u\,\frac{d}{du}\left[\arctan{\left(\frac{u-r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}-\arctan{\left(\frac{u+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\right]\\ &~~~~~-\int_{r^{2}}^{1}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\arctan{\left(u\right)}\\ &=\frac{\pi}{2}\bigg{[}\arctan{\left(\frac{1-r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}-\arctan{\left(\frac{1+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\\ &~~~~~-\arctan{\left(\frac{1-\cos{\left(\sigma\right)}}{\sin{\left(\sigma\right)}}\right)}+\arctan{\left(\frac{1+\cos{\left(\sigma\right)}}{\sin{\left(\sigma\right)}}\right)}\bigg{]}\\ &~~~~~+\int_{r^{2}}^{1}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)}\\ &~~~~~-\int_{r^{2}}^{1}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)}\\ &=\frac{\pi}{2}\left[\frac{\pi}{2}-\sigma-\arctan{\left(\frac{r^{2}\sin{\left(2\sigma\right)}}{1-r^{2}\cos{\left(2\sigma\right)}}\right)}\right]\\ &~~~~~+\int_{r^{2}}^{1}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)}\\ &~~~~~+\int_{-1}^{-r^{2}}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)};~~~\small{\left[u\mapsto-u\right]}\\ &=\frac{\pi}{2}\left[\frac{\pi}{2}-\sigma-\arctan{\left(\frac{r^{2}\sin{\left(2\sigma\right)}}{1-r^{2}\cos{\left(2\sigma\right)}}\right)}\right]\\ &~~~~~+\int_{-1}^{1}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)}\\ &~~~~~-\int_{-r^{2}}^{r^{2}}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)}.\tag{7}\\ \end{align}$$

To facilitate the evaluation of the remaining two integrals in the last line above, we introduce yet another auxiliary function $\mathcal{K}:\left(0,1\right)\times\left(0,\frac{\pi}{2}\right)\times\left(0,1\right]\rightarrow\mathbb{R}$ defined via the definite integral

$$\mathcal{K}{\left(r,\sigma,z\right)}:=\int_{-z}^{z}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)}.\tag{8}$$

Suppose $\left(r,\sigma,z\right)\in\left(0,1\right)\times\left(0,\frac{\pi}{2}\right)\times\left(0,1\right]$. We then have

$$\begin{align} \mathcal{K}{\left(r,\sigma,z\right)} &=\int_{-z}^{z}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)}\\ &=\int_{-z}^{z}\mathrm{d}u\,\arctan{\left(u\right)}\frac{d}{du}\arctan{\left(\frac{u+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\\ &=\arctan{\left(z\right)}\arctan{\left(\frac{z+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}-\arctan{\left(-z\right)}\arctan{\left(\frac{-z+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\\ &~~~~~-\int_{-z}^{z}\mathrm{d}u\,\arctan{\left(\frac{u+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\frac{d}{du}\arctan{\left(u\right)};~~~\small{I.B.P.}\\ &=\arctan{\left(z\right)}\arctan{\left(\frac{z+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}-\arctan{\left(z\right)}\arctan{\left(\frac{z-r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\\ &~~~~~-\int_{-z}^{z}\mathrm{d}u\,\frac{1}{1+u^{2}}\arctan{\left(\frac{r\cos{\left(\sigma\right)}+u}{r\sin{\left(\sigma\right)}}\right)}\\ &=\arctan{\left(z\right)}\left[\arctan{\left(\frac{z+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}-\arctan{\left(\frac{z-r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\right]\\ &~~~~~-\int_{-z}^{z}\mathrm{d}u\,\frac{1}{1+u^{2}}\arctan{\left(\cot{\left(\sigma\right)}+r^{-1}u\csc{\left(\sigma\right)}\right)}\\ &=\arctan{\left(z\right)}\left[\arctan{\left(\frac{z+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}-\arctan{\left(\frac{z-r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\right]\\ &~~~~~-\int_{-\arctan{\left(z\right)}}^{\arctan{\left(z\right)}}\mathrm{d}\varphi\,\arctan{\left(\cot{\left(\sigma\right)}+r^{-1}\csc{\left(\sigma\right)}\tan{\left(\varphi\right)}\right)};~~~\small{\left[\arctan{\left(u\right)}=\varphi\right]}.\tag{9}\\ \end{align}$$

Suppose $\left(a,b,\psi,\omega\right)\in\mathbb{R}\times\mathbb{R}\times\left(-\frac{\pi}{2},\frac{\pi}{2}\right)\times\left(-\frac{\pi}{2},\frac{\pi}{2}\right)$, and assume

$$0<a\land\sqrt{1+a^{2}}<b\land-\frac{\pi}{4}\le\psi<\omega\le\frac{\pi}{4}.$$

Set

$$B:=\frac{\left[\sqrt{a^{2}+\left(b+1\right)^{2}}+\sqrt{a^{2}+\left(b-1\right)^{2}}\right]^{2}}{4b}\land\phi:=\frac12\arctan{\left(\frac{2ab}{b^{2}-a^{2}-1}\right)},$$

and note that $1<B\land0<\phi<\frac{\pi}{4}$. Then, it can be shown that

$$\begin{align} \int_{\psi}^{\omega}\mathrm{d}\varphi\,\arctan{\left(a+b\tan{\left(\varphi\right)}\right)} &=\frac12\left(\phi+\omega\right)^{2}-\frac12\left(\phi+\psi\right)^{2}\\ &~~~~~-\left(\omega-\psi\right)\arctan{\left(\frac{\tan{\left(\phi\right)}}{B}\right)}\\ &~~~~~-\frac12\operatorname{Li}_{2}{\left(\frac{1-B}{1+B},\pi-2\phi-2\omega\right)}\\ &~~~~~+\frac12\operatorname{Li}_{2}{\left(\frac{1-B}{1+B},\pi-2\phi-2\psi\right)}.\tag{10}\\ \end{align}$$

The integration formula above is derived in this question.

Integration formula $(10)$ is sufficient to provide us with a closed-form expression for $\mathcal{K}$, and in turn $\mathcal{J}$ and $\mathcal{I}$ as well. So our work is complete in principle, and all that remains is to substitute back the chain of results to obtain a final expression in terms of the original variables. (Forgive me if I don't bother to do that here since the end result is such a cumbersome expression.)