How to prove $\int_0^\infty J_\nu(x)^3dx\stackrel?=\frac{\Gamma(1/6)\ \Gamma(1/6+\nu/2)}{2^{5/3}\ 3^{1/2}\ \pi^{3/2}\ \Gamma(5/6+\nu/2)}$?

I am interested in finding a general formula for the following integral: $$\int_0^\infty J_\nu(x)^3dx,\tag1$$ where $J_\nu(x)$ is the Bessel function of the first kind: $$J_\nu(x)=\sum _{n=0}^\infty\frac{(-1)^n}{\Gamma(n+1)\Gamma(n+\nu+1)}\left(\frac x2\right)^{2n+\nu}.\tag2$$ Mathematica gives the following result: $$\int_0^\infty J_\nu(x)^3dx=\frac1{\pi\,\nu}{_3F_2}\left(\frac12,\frac12-\nu,\frac12+\nu;1-\frac\nu2,1+\frac\nu2;\frac14\right)+\frac{\Gamma\left(-\frac\nu2\right)\Gamma\left(\frac{1+3\nu}2\right)\cos\frac{\pi\,\nu}2}{2^{\nu+1}\ \pi^{3/2}\ \Gamma(\nu+1)}{_3F_2}\left(\frac{1-\nu}2,\frac{1+\nu}2,\frac{1+3\nu}2;1+\frac\nu2,1+\nu;\frac14\right),\tag3$$ which can be significantly simplified for odd and half-integer values of $\nu$. The results at those points allow to conjecture another, simpler general formula: $$\int_0^\infty J_\nu(x)^3dx\stackrel?=\frac{\Gamma\left(\frac16\right)\ \Gamma\left(\frac16+\frac\nu2\right)}{2^{5/3}\ 3^{1/2}\ \pi^{3/2}\ \Gamma\left(\frac56+\frac\nu2\right)},\tag4$$ which agrees with $(3)$ to a very high precision for many different values of $\nu$. It also has an advantage that it is defined for all $\nu>-\frac13$ whereas $(3)$ is undefined at every even $\nu$ and requires calculating a limit at those points.

Is it possible to prove the formula $(4)$?

Thank you for posting this question, I enjoyed trying to answer it.

Start with the expression that Mathematica gave you and replace each argument $\frac14$ of a hypergeometric function with $\frac z4$, because we will be taking limits. I will call the two hypergeometric functions $Q_1(z)$ and $Q_2(z)$. Each term can be brought to a closed form by using identity 16.6.2 from the DLMF.

Setting $a=\frac12$, $b=1-\frac\nu2$, we get $$ Q_1(z) = (1-z)^{-\frac12} F\left(\frac16, \frac36,\frac 56; 1-\frac\nu2, 1+\frac\nu2; \frac{-27 z}{4(1-z)^3} \right), $$ and setting $a=\frac{1+3\nu}{2}$, $b=1+\frac\nu2$, we get $$ Q_2(z) = (1-z)^{-\frac{1+3\nu}{2}} F\left(\frac{1+3\nu}6, \frac{3+3\nu}{6}, \frac{5+3\nu}{6}; 1+\frac\nu2, 1+\nu; \frac{-27z}{4(1-z)^3} \right). $$ (Note that there are 6 possible identities to try per function, one for each possible choice of $a$ and $b$ from the parameters, so it helps to do this on a computer.)

The reason this works is that now the point $z=1$ is a singular point of the hypergeometric functions on the right hand side, and Mathematica will succeed in finding the limits as $z\to1$. The expression for the whole integral that you have is $$ Q = \frac{2^{\frac43}\pi^{\frac12}}{\Gamma(-\frac16)\Gamma(\frac56+\frac\nu2)\Gamma(\frac56-\frac\nu2)\sin\frac{\nu\pi}2} \left( -1 + 3^{-\frac{3\nu}2}\cos\left(\frac{\nu\pi}{2}\right) \frac{\Gamma(\frac{1+3\nu}{2}) \Gamma(\frac56-\frac\nu2)}{\Gamma(\frac{1+\nu}2)\Gamma(\frac56+\frac\nu2)} \right). $$ Call the large expression in brackets $A$, and then write $$ A = -1 + B 3^{-\frac{3\nu}{2}}\cos\frac{\pi\nu}{2}, \qquad B = \frac{\Gamma(\frac{1+3\nu}{2}) \Gamma(\frac56-\frac\nu2)}{\Gamma(\frac{1+\nu}2)\Gamma(\frac56+\frac\nu2)}. $$

Now, Mathematica will not simplify $A$ or $B$ on its own, so it needs help. Set $x=\frac16+\frac\nu2$, and use the multiplication formula to get $$ \frac{\Gamma(\frac{1+3\nu}2)}{\Gamma(\frac{1+\nu}{2})\Gamma(\frac56+\frac\nu2)} = \frac{\Gamma(3x)}{\Gamma(x+\frac13)\Gamma(x+\frac23)} = \frac{\Gamma(x)}{2\pi} 3^{3x-1/2}. $$ After this, $A$ simplifies to $$ A = -1 + \frac{\Gamma(\frac16+\frac\nu2)\Gamma(\frac56-\frac\nu2)}{2\pi}\cos\frac{\pi\nu}{2} = -1 + \frac{\cos\frac{\pi\nu}{2}}{2\sin(\frac\pi6+\frac{\pi\nu}{2})}, $$ where I've also used the reflection formula for $\Gamma(z)\Gamma(1-z)$ to get rid of the gamma functions. Some further amount of manual trigonometry yields $$ A = -\frac{\sqrt{3}}{2}\frac{\sin\frac{\pi\nu}{2}}{\sin(\frac\pi6 + \frac{\nu\pi}{2})}. $$

Finally, write $$ \frac{1}{\sin(\frac\pi6+\frac{\pi\nu}{2})} = \frac{\Gamma(\frac16+\frac\nu2)\Gamma(\frac56-\frac\nu2)}{\pi}, $$ and substitute back. Lots of things cancel, and the answer is $$ Q = -\frac{3^{1/2}2^{1/3}}{\pi^{1/2}} \frac{\Gamma(\frac16+\frac\nu2)}{\Gamma(-\frac16)\Gamma(\frac56+\frac\nu2)}. $$

This closed form is equivalent to the one you gave through the use of $\Gamma(\frac16)\Gamma(-\frac16)=-12\pi$.

P.S. I would also like to note that the integral $$ I(\nu,c) = \int_0^\infty J_\nu(x)^2 J_\nu(c x)\,dx $$ and its general form $$ \int_0^\infty x^{\rho-1}J_\nu(a x) J_\mu(b x) J_\lambda(c x)\,dx $$ appear in Gradshteyn and Ryzhik, and you can find a paper "Some infinite integrals involving bessel functions, I and II" by W. N. Bailey, which evaluates this integral in terms of Appell functions, but only in the case $c>2$ ($|c|>|a|+|b|$), which is where the $F_4$ Appell function converges. DLMF 16.16.6 actually gives a way to write this integral as $$ \frac{\Gamma(\frac{1+3\nu}{2})c^{-1-2\nu}}{\Gamma(1+\nu)^2\Gamma(\frac{1-\nu}{2})} \,\,\,{}_2F_1\left( \frac{1+\nu}{2}, \frac{1+3\nu}{2}; 1+\nu; x \right)^2, \qquad x = \frac{1-\sqrt{1-4/c^2}}{2}, $$ but the issue is that this is only correct for $c>2$, and the rhs is complex for $c<2$. Appell function would only be defined by analytic continuation in this case anyway, and I didn't find anything useful about non-principal branches of Appell or hypergeometric functions.

For $c>2$, Mathematica also gives the following: $$ I(\nu,c) = \frac{\Gamma(\frac{1+3\nu}{2})c^{-1-2\nu}}{\Gamma(\frac{1-\nu}{2})\Gamma(1+\nu)^2} \,\,\,{}_3F_2\left( \frac{1+\nu}{2}, \frac{1}{2}+\nu, \frac{1+3\nu}{2}; 1+\nu, 1+2\nu; \frac{4}{c^2} \right), $$ but this is incorrect when $c<2$.

See section 6 in http://arxiv.org/abs/1007.0667 for that type of triple-product integrals.