Proving $\int_{\sqrt{5/7}}^1 \frac{(\pi-3\arctan\sqrt{\frac{2x^2-1}{3x^2-2}})\arctan x}{\sqrt{2x^2-1}(3x^2-1)} dx = \frac{\pi^3}{672}$

Half a year ago I posted a problem here, in which a remarkable result is proved by a unusual (and nontrivial) approach: $$\int_{\sqrt{\frac{3}{5}}}^1 \frac{\arctan (x)}{\sqrt{2 x^2-1} \left(3 x^2-1\right)} \, dx=\frac{3 \pi ^2}{160}$$ I collected it from a Ramanujan-like blog where produce only formulas but no proofs. If fact, the blogger states a harder version of this kind of 'pathological' integral: $$\pi \int_{\sqrt{5/7}}^1 \frac{\arctan y}{\left(3 y^2-1\right)\sqrt{2 y^2-1}} \, dy-3 \int_{\sqrt{5/7}}^1 \frac{\arctan (y) \arctan\sqrt{\frac{2 y^2-1}{3 y^2-2}}}{\left(3 y^2-1\right) \sqrt{2 y^2-1}} \, dy=\frac{\pi ^3}{672}$$ My question is: how to prove the second identity? Even armed with weapons offered in the link, I haven't found a possible way. Any kind of help will be appreciated.

Let $I$ denotes the integral, then $$I = \frac{1}{2}\int_{5/7}^1 {\frac{{\arctan \sqrt x }}{{\sqrt x (3x - 1)\sqrt {2x - 1} }}(\pi - 3\arctan \sqrt {\frac{{2x - 1}}{{3x - 2}}} )dx} $$ let $x = \frac{{3 + u}}{{5 + u}}$, then $\arctan\sqrt{{u^2} - 1} = \pi - 2\arctan \sqrt {\frac{{2x - 1}}{{3x - 2}}}$, so $$I = \frac{1}{4}\int_2^\infty {\frac{{\arctan \sqrt {\frac{{3 + u}}{{5 + u}}} }}{{\sqrt {1 + u} (2 + u)\sqrt {3 + u} }}(3\arctan \sqrt {{u^2} - 1} - \pi )du} $$ Note that, for $u>2$, $$\int_{u/2}^{u - 1} {\frac{{dv}}{{\sqrt {u - v} \sqrt v (1 + u - v)(1 + v)}}} = \frac{{3\arctan \sqrt {{u^2} - 1} - \pi }}{{\sqrt {1 + u} (2 + u)}}$$ so $$I = \frac{1}{4}\int_2^\infty {\int_{u/2}^{u - 1} {\frac{{\arctan \sqrt {\frac{{3 + u}}{{5 + u}}} }}{{\sqrt {3 + u} }}\frac{1}{{\sqrt {u - v} \sqrt v (1 + u - v)(1 + v)}}} dvdu} $$ change of variables $u=x+y, v=y$ gives $$\begin{aligned}I &= \frac{1}{4}\int_1^\infty {\int_x^\infty {\frac{{\arctan \sqrt {\frac{{3 + x + y}}{{5 + x + y}}} }}{{\sqrt {3 + x + y} }}\frac{1}{{\sqrt x \sqrt y (1 + x)(1 + y)}}} dxdy} \\ &= \frac{1}{2}\int_1^\infty {\int_1^\infty {\frac{{\arctan \sqrt {\frac{{3 + {x^2} + {y^2}}}{{5 + {x^2} + {y^2}}}} }}{{\sqrt {3 + {x^2} + {y^2}} }}\frac{1}{{(1 + {x^2})(1 + {y^2})}}} dxdy} \qquad \text{(By symmetry)} \\ &= \frac{1}{2}\int_1^\infty {\int_1^\infty {\int_0^1 {\frac{1}{{\sqrt {4 + {x^2} + {y^2} + {z^2}} }}} \frac{{dxdydz}}{{(1 + {x^2})(1 + {y^2})(1 + {z^2})}}} } = \frac{f(1,2)}{2} \end{aligned}$$ where for $n_1,n_2\geq 0$, $n=n_1+n_2$, $f(n_1,n_2)$ is the $n$-dimensional integral, $$f(n_1,n_2) = \int_{{{[0,\pi /4]}^{{n_1}}}{{\times [\pi /4,\pi /2]}^{n_2}}} {\frac{1}{{{{(1 + {{\sec }^2}{x_1} + ... + {{\sec }^2}{x_n})}^{1/2}}}}d{x_i}} $$

I will show that $f(n_1,n_2)/\pi^n \in \mathbb{Q}$, and give a recurrence formula for it, from which $\color{red}{f(1,2) = \frac{\pi^3}{336}}$ is obtained, thereby completing the proof.

(Lemma) Let $n_1,n_2$ be nonnegative integers, $n=n_1+n_2$, $m,r>0$. If $mr=n+1$, then $$\int_{{{[0,1]}^{{n_1}}}{{\times[0,\infty ]}^{{n_2}}}} {\frac{1}{{{{(1 + {x_1}^r + ... + {x_n}^r)}^m}}}d{x_i}} = \frac{r}{{\Gamma (m)}}\frac{{\Gamma {{(1 + \frac{1}{r})}^{n + 1}}}}{{{n_1} + 1}} $$

Proof: Let $f(x) = \int_0^x e^{-t^r}dt$, then $$\begin{aligned} &\quad\int_{{{[0,1]}^{{n_1}}}{{\times [0,\infty ]}^{{n_2}}}} {\frac{1}{{{{(1 + {x_1}^r + ... + {x_n}^r)}^m}}}d{x_i}}\\ &= \frac{1}{{\Gamma (m)}}\int_0^\infty {\int_{{{[0,1]}^{{n_1}}}{{[0,\infty ]}^{{n_2}}}} {{t^{m - 1}}{e^{ - (1 + {x_1}^r + ... + {x_n}^r)t}}dt} } \\ &= \frac{1}{{\Gamma (m)}}\int_0^\infty {{t^{m - 1}}{e^{ - t}}{{\left( {\int_0^\infty {{e^{ - {x^r}t}}dx} } \right)}^{{n_2}}}{{\left( {\int_0^1 {{e^{ - {x^r}t}}dx} } \right)}^{{n_1}}}dt} \\ &= \frac{{f{{(\infty )}^{{n_2}}}}}{{\Gamma (m)}}\int_0^\infty {{t^{m - 1}}{t^{ - n/r}}{e^{ - t}}{{\left( {\int_0^{{t^{1/r}}} {{e^{ - {x^r}}}dx} } \right)}^{{n_1}}}dt} \\ &= \frac{{\Gamma {{(1 + \frac{1}{r})}^{{n_2}}}r}}{{\Gamma (m)}}\int_0^\infty {{t^{mr - n - 1}}{e^{ - {t^r}}}f{{(t)}^{{n_1}}}dt} \end{aligned}$$ if $mr=n+1$, then the antiderivative of integrand is $f(x)^{n_1+1}/(n_1+1)$, the result follows. QED

Now let $$\begin{aligned}S &= \{(x,y)\subset \mathbb{R}^2 | 0\leq x,y\leq 1\} \\ T &= \{(x,y)\subset \mathbb{R}^2 | 0\leq y\leq x\leq 1\} \\ R &= \{(x,y)\subset \mathbb{R}^2 | 0\leq x \leq 1, y\geq x\} \\ U &= \{(x,y)\subset \mathbb{R}^2 | 0\leq x \leq 1, y\geq 0\} \end{aligned}$$ note that under polar coordinates, $T$ and $R$ correspond to $0\leq r \leq \sec \theta, 0\leq \theta \leq \pi/4$ and $0\leq r \leq \sec \theta, \pi/4 \leq \theta \leq \pi/2$ respectively. For any (measurable) set $A$, let $$m(A) = \int_{A} \frac{dx_i}{(1+x_1^2+\cdots+x_{2n}^2)^{(2n+1)/2}} $$ this is symmetric under permutation of each $2n$ coordinates, consider ($n=n_1+n_2$) $$\begin{aligned}m({T^{{n_1}}} \times {R^{{n_2}}}) &= m({T^{{n_1}}} \times {(U - T)^{{n_2}}}) \\ & = \sum\limits_{k = 0}^{{n_2}} {\binom{n_2}{k}{{( - 1)}^k}m({T^{{n_1} + k}} \times {U^{{n_2} - k}})} = \sum\limits_{k = 0}^{{n_2}} {\binom{n_2}{k}\frac{{{{( - 1)}^k}}}{{{2^{{n_1} + k}}}}m({S^{{n_1} + k}} \times {U^{{n_2} - k}})} \\ & = \sum\limits_{k = 0}^{{n_2}} {\binom{n_2}{k}\frac{{{{( - 1)}^k}}}{{{2^{{n_1} + k}}}}m({{[0,1]}^{2{n_1} + {n_2} + k}} \times {{[0,\infty ]}^{{n_2} - k}})} \\ \end{aligned}$$ Lemma implies $$\tag{1}m({T^{{n_1}}} \times {R^{{n_2}}}) = \frac{{\Gamma {{(\frac{3}{2})}^{2n + 1}}}}{{\Gamma (\frac{{2n + 1}}{2})}}\sum\limits_{k = 0}^{{n_2}} {\binom{n_2}{k}\frac{{{{( - 1)}^k}}}{{{2^{{n_1} + k}}}}\frac{2}{{2{n_1} + {n_2} + k + 1}}}$$ On the other hand, polar coordinates give, by integrating all $r_i$, $$\begin{aligned}m({T^{{n_1}}} \times {R^{{n_2}}}) &= \int_{{{[0,\sec {\theta _i}]}^n} \times {{[0,\pi /4]}^{{n_1}}} \times {{[\pi/4,\pi /2]}^{{n_2}}}} {\frac{{{r_1}...{r_n}d{r_i}d{\theta _i}}}{{{{(1 + {r_1}^2 + ... + {r_n}^2)}^{(2n + 1)/2}}}}}\\ &=\frac{1}{{(2n - 1)(2n - 3)...(1)}} {\sum\limits_{i,j \ge 0} {{{(\frac{\pi }{4})}^{{n_1} + {n_2} - i - j}}{{( - 1)}^{i + j}}\binom{n_1}{i}\binom{n_2}{j}f(i,j)} } \end{aligned}$$ Compare this with $(1)$, denoting $f(i,j) = (\pi/4)^{i+j} \tilde{f}(i,j)$ gives $$\tag{2}{2^{n - 1}}\sum\limits_{k = 0}^{{n_2}} {\binom{n_2}{k}\frac{{{{( - 1)}^k}}}{{{2^{{n_1} + k}}}}\frac{2}{{2{n_1} + {n_2} + k + 1}}} = \sum\limits_{i,j \ge 0} {{{( - 1)}^{i + j}}\binom{n_1}{i}\binom{n_2}{j}\tilde{f}(i,j)} $$

This is our desired recurrence, starting with $f(0,0)=1, f(1,0)=\pi/6, f(0,1)=\pi/12$, we can computes $f(i,j)$ for $i+j=2$, for example, letting $n_1=1, n_2=1$ in $(2)$ gives $f(1,1)$. The following are values of $f(i,j)$ for $i+j\leq 3$: $$\begin{aligned}&f(0,0)=1 \\ &f(1,0)=\pi/6 \quad f(0,1)=\pi/12 \\ &f(2,0)=\pi^2/30\quad f(1,1)=3\pi^2/160\quad f(0,2)=\pi^2/80\\ &f(3,0)=\pi^2/140\quad f(2,1)=29\pi^3/6720\quad \color{red}{f(1,2)=\pi^3/336}\quad f(0,3)=\pi^3/448 \\ \end{aligned}$$

Remarks:

• It can be shown that $f(n,0)=\frac{n!^2}{(2n+1)!}\pi^n$.

• Mathematica code to calculate $f(n_1,n_2)$:

int[n10_Integer, n20_Integer] /; Min[n10, n20] >= 0 := Block[{n1 = n10, n2 = n20, f, n, g, m, r}, n = n1 + n2; f[0, 0] = 1; g[a_, b_] := 2^(a + b - 1)* Sum[Binomial[b, k] (-1)^k/2^(a + k)*2/(2 a + b + k + 1), {k, 0, b}]; For[m = 1, m <= n, m++, For[r = 0, r <= m, r++, f[r, m - r] = (-1)^ m*(g[r, m - r] - Plus @@ ((-1)^(#1 + #2)*Binomial[r, #1]*Binomial[m - r, #2]* f[#1, #2] & @@@ Most@Flatten[Table[{i, j}, {i, 0, r}, {j, 0, m - r}], 1]))]]; f[n1, n2]*(Pi/4)^n];

For example, int[12,6] gives $f(12,6) = \frac{807986899 \pi ^{18}}{795635388421609881600}$.

• Consider $f(1,1)$, letting $y = \sqrt{(2+x^2)/(4+x^2)}$ gives $$f(1,1)=\frac{3\pi^2}{160}=\int_1^\infty {\frac{{\arctan \sqrt {\frac{{2 + {x^2}}}{{4 + {x^2}}}} }}{{(1 + {x^2})\sqrt {2 + {x^2}} }}dx} = \int_{\sqrt {3/5} }^1 {\frac{{\arctan y}}{{(3{y^2} - 1)\sqrt {2{y^2} - 1} }}dy} $$ This is a more direct proof of this non-trivial result, without using Schläfli's $S(\alpha,\beta,\gamma)$. The current question can be seen as a higher-dimensional analogue of such already difficult result.