Scary contour integral, but is also an integral representation for $\Gamma$-function

This problem is supposed to be from an old Acta Mathematica volume I circa 1880's, and is attributed to Bourguet.

By using a parabola with its focus on the origin as a contour, show that:

$$\int_{0}^{\infty}e^{-ax^{2}}(1+x^{2})^{z-1/2}\cos(2ax+(2z-1)\tan^{-1}(x))dx=\frac{\sin(\pi z)\Gamma(z)}{2a^{z}e^{a}}$$

I am not a total cabbage head with contour integration, yet certainly not along the lines of Ron G, RV, robjohn, et. al.

But, this thing looks nasty. Sometimes looks can be deceiving.

I have never used a parabola as a contour. The closest thing I can think of is a Hankel contour. That is not exactly a parabola, but is commonly used with the Gamma function integral. A Hankel looks more like a bobby pin. But, if anyone would care to exert their contour integration prowess, it would be nice to see this one worked out.

Let $\mathcal{I}$ be the nasty integral we want to calculate. For $x \in \mathbb{R}$, we have

$$e^{\pm i\tan^{-1}x} = \frac{1\pm i x}{\sqrt{1+x^2}}$$

This leads to $$\cos(2ax + (2z-1)\tan^{-1}x) = \frac12\left[ e^{2iax} \left(\frac{1 + ix}{\sqrt{1+x^2}}\right)^{2z-1} + e^{-2iax} \left(\frac{1-ix}{\sqrt{1+x^2}}\right)^{2z-1} \right] $$ As a result, we can rewrite integral $\mathcal{I}$ as

$$\begin{align} \mathcal{I} &= \frac{1}{2e^a}\int_0^\infty \left( e^{a(1+ix)^2} (1+ix)^{2z-1} + e^{a(1-ix)^2}(1-ix)^{2z-1}\right) dx\\ &= \frac{1}{2i e^a} \int_{1-i\infty}^{1+i\infty} e^{at^2} t^{2z-1}dt\\ &= \frac{1}{4i e^a a^z} \int_C e^t t^{z-1} dt \end{align}$$ where $C$ is the contour $\mathbb{R} \ni x \mapsto a(1+ix)^2 \in \mathbb{C}$. It is easy to see $C$ is a parabola which start from infinity at third quadrant. It first move towards the origin, circle around it counterclockwisely in the fourth and then first quadrant. Finally, move away to infinity in the second quadrant.

To evaluate $\mathcal{I}$, we can deform $C$ to a keyhole contour along the negative axis. The keyhole contour start at $-\infty -i\epsilon$, move towards the $-i\epsilon$, circle around the origin counterclockwisely to $i\epsilon$ and then move away to $-\infty + i\epsilon$. This give us $$\mathcal{I} = \frac{1}{4i e^a a^z}\int_C e^{t} t^{z-1} dt = \frac{ e^{i\pi z} - e^{-i\pi z} }{4i e^a a^z}\Gamma(z) = \frac{\sin(\pi z)}{2 e^a a^z}\Gamma(z)$$

This answer is along the same lines as achille hui's with subtle differences, but I've tried to provide more detail. Let's start with $$ \begin{align} &\cos(2ax+(2z-1)\tan^{-1}(x))\\ &=\frac12\left[e^{i2ax+i(2z-1)\tan^{-1}(x)}+e^{-i2ax-i(2z-1)\tan^{-1}(x)}\right]\\ &=\frac12\left[e^{i2ax}\left(\frac{1+ix}{\sqrt{1+x^2}}\right)^{2z-1}+e^{-i2ax}\left(\frac{1-ix}{\sqrt{1+x^2}}\right)^{2z-1}\right]\\ &=\mathrm{Re}\left[e^{i2ax}\left(\frac{1+ix}{\sqrt{1+x^2}}\right)^{2z-1}\right]\tag{1} \end{align} $$ which follows because $$ \begin{align} e^{\pm i\tan^{-1}(x)} &=\cos(\tan^{-1}(x))\pm i\sin(\tan^{-1}(x))\\ &=\frac{1\pm ix}{\sqrt{1+x^2}}\tag{2} \end{align} $$ Therefore, $$ \begin{align} &\int_0^\infty e^{-ax^2}(1+x^2)^{z-1/2}\cos(2ax+(2z-1)\tan^{-1}(x))\,\mathrm{d}x\tag{3}\\ &=\frac12\int_{-\infty}^\infty e^{-ax^2}(1+x^2)^{z-1/2}\cos(2ax+(2z-1)\tan^{-1}(x))\,\mathrm{d}x\tag{4}\\ &=\frac12\mathrm{Re}\left[\int_{-\infty}^\infty e^{-ax^2}e^{i2ax}(1+ix)^{2z-1}\,\mathrm{d}x\right]\tag{5}\\ &=\frac12\mathrm{Re}\left[e^{-a}e^{i\pi(z-1/2)}\int_{-\infty}^\infty e^{-a(x-i)^2}(x-i)^{2z-1}\,\mathrm{d}x\right]\tag{6}\\ &=\frac14\mathrm{Re}\left[e^{-a}e^{i\pi(z-1/2)}\int_{\gamma_1} e^{-aw}w^{z-1}\,\mathrm{d}w\right]\tag{7}\\ &=\frac14\mathrm{Re}\left[e^{-a}e^{i\pi(z-1/2)}\int_{\gamma_2} e^{-aw}w^{z-1}\,\mathrm{d}w\right]\tag{8}\\ &=\frac14\mathrm{Re}\left[e^{-a}e^{i\pi(z-1/2)}(1-e^{-2\pi iz})\int_0^\infty e^{-at}t^{z-1}\,\mathrm{d}t\right]\tag{9}\\ &=\frac12\mathrm{Re}\left[e^{-a}\sin(\pi z)\,a^{-z}\Gamma(z)\right]\tag{10}\\ &=\frac{\sin(\pi z)\Gamma(z)}{2\,a^ze^a}\tag{11} \end{align} $$ Explanation:
$\:\ (4)$: the integrand is even so duplicate the domain and divide by $2$
$\:\ (5)$: apply $(1)$
$\:\ (6)$: complete the square in the exponential and factor out $e^{i\pi(z-1/2)}$
$\:\ (7)$: substitute $w=(x-i)^2$: $\gamma_1$ is the counterclockwise parabola $x=\frac14y^2-1$
$\:\ (8)$: change the contour: $\gamma_2$ is $[\infty,0]+i\epsilon\cup i\epsilon[1,-1]\cup[0,\infty]-i\epsilon$
$\:\ (9)$: substitute $w=t$ along the bottom of $[0,\infty]$ and $w=e^{-2\pi i}t$ along the top
$(10)$: use the definition of $\sin(z)$ and $\Gamma(z)$
$(11)$: apply $\mathrm{Re}$

Make use of $\Gamma(z) \sin(\pi z) = \dfrac{\pi}{\Gamma(1-z)}$ and the integral representation of reciprocal of $\Gamma(z)$. $$\dfrac1{\Gamma(1-z)} = \dfrac{i}{2\pi} \displaystyle \oint_C \dfrac{e^{-t}}{(-t)^{1-z}}dt$$ where $C$ encloses the origin. The rest is some brute-force brainless algebra.