Symmetry of function defined by integral

Define a function $f(\alpha, \beta)$, $\alpha \in (-1,1)$, $\beta \in (-1,1)$ as

$$ f(\alpha, \beta) = \int_0^{\infty} dx \: \frac{x^{\alpha}}{1+2 x \cos{(\pi \beta)} + x^2}$$

One can use, for example, the Residue Theorem to show that

$$ f(\alpha, \beta) = \frac{\pi \sin{\left (\pi \alpha \beta\right )}}{ \sin{\left (\pi \alpha\right )} \, \sin{\left (\pi \beta\right )}} $$

Clearly, from this latter expression, $f(\alpha, \beta) = f(\beta, \alpha)$. My question is, can one see this symmetry directly from the integral expression?

Very interesting question! But, alas, not an answer. Only few representations for the integral obtained. One of them evaluated to the form claimed in the question.

First, transform the integral into a form, symmetric under $\alpha \mapsto -\alpha$: $$ \int_0^\infty \frac{x^\alpha}{1+2 x \cos(\pi \beta) + x^2} \mathrm{d} x = \int_0^1 \frac{x^\alpha}{1+2 x \cos(\pi \beta) + x^2} \mathrm{d} x + \int_1^\infty \frac{x^\alpha}{1+2 x \cos(\pi \beta) + x^2} \mathrm{d} x $$ Make a change of variables $x \to x^{-1}$ in the last integral to obtain: $$ f(\alpha,\beta) = \int_0^1 \frac{x^\alpha + x^{-\alpha}}{1+2 x \cos(\pi \beta) + x^2} \mathrm{d} x \tag{1} $$ Now, making a change of variables $x = \exp(-t)$ we have: $$ f(\alpha,\beta) = \int_0^\infty \frac{\cosh(\alpha t)}{\cosh(t) + \cos(\beta \pi)} \mathrm{d} t \tag{2} $$ Using $$ \int_0^\infty \exp\left(-u \left( \cosh t + \cos \pi \beta \right) \right) \mathrm{d}u = \frac{1}{\cosh(t) + \cos(\beta \pi)} $$ and the integral representation of the modified Bessel function of the second kind: $$ \int_0^\infty \cosh(\alpha t) \exp\left( - u \cosh t \right) \mathrm{d}t = K_\alpha(u) $$ we arrive at a compact representation: $$ f(\alpha,\beta) = \int_0^\infty K_\alpha(u) \mathrm{e}^{-u \cos\left(\pi \beta\right)} \mathrm{d} u \tag{3} $$ expanding the exponential into series and using $\int_0^\infty u^n K_\alpha(u) \mathrm{d} u = 2^{n-1} \Gamma\left(\frac{n}{2} + \frac{1+\alpha}{2} \right)\Gamma\left(\frac{n}{2} + \frac{1-\alpha}{2} \right)$ we get: $$ f(\alpha,\beta) = \sum_{n=0}^\infty \frac{2^{n-1}}{n!} \left(-\cos \pi \beta\right)^{n} \Gamma\left(\frac{n}{2} + \frac{1+\alpha}{2} \right)\Gamma\left(\frac{n}{2} + \frac{1-\alpha}{2} \right) \tag{4} $$ summing over even and over odd integers: $$ f(\alpha, \beta) = \frac{\pi}{2} \frac{ \cos\left( \alpha \arcsin \cos(\pi \beta) \right) }{ | \sin(\pi \beta) | \cos \left( \frac{\pi \alpha}{2} \right)} - \frac{\pi}{2} \frac{ \sin\left( \alpha \arcsin \cos(\pi \beta) \right) }{ | \sin(\pi \beta) | \sin \left( \frac{\pi \alpha}{2} \right)} = \pi \frac{\sin \left( \alpha \left( \frac{\pi}{2} - \arcsin \cos(\pi \beta) \right) \right)}{ | \sin \pi \beta | \sin(\pi \alpha)} $$ Now $\frac{\pi}{2} - \arcsin \cos(\pi \beta) = \arccos \cos(\pi \beta) = \pi | \beta |$ for $-1<\beta<1$. Thus, restoring parity, we recover the OP's expression: $$ f(\alpha, \beta) = \pi \frac{ \sin(\pi \alpha \beta)}{\sin(\pi \alpha) \sin(\pi \beta)} = \frac{\operatorname{sinc}(\pi \alpha \beta)}{\operatorname{sinc}(\pi \alpha) \operatorname{sinc}(\pi \beta)} \tag{5} $$

This question is fantastic! I have not found an integral displaying the symmetry but I wanted to show that it points to a symmetry of the Lerch zeta function, if this is known then perhaps that could explain the symmetry, if it is not known it is very interesting I think.

From $(1)$ in @Sasha: $$\begin{align} f(\alpha,\beta) = \int_0^1 \frac{x^\alpha + x^{-\alpha}}{1+2 x \cos(\pi \beta) + x^2} \mathrm{d} x \tag{1}\\ \end{align} $$ The integrand is comparable to the generating function of the Chebyshev polynomials of the second kind and in fact: $$ \begin{align} {\frac {{x}^{\alpha}+{x}^{-\alpha}}{1+2\,x\cos \left( \pi \,\beta \right) +{x}^{2}}}&=\sum _{n=0}^{\infty }U_n \left( - \cos \left( \pi \,\beta \right) \right) \left( {x}^{n+\alpha}+{x}^{n -\alpha} \right) \tag{2}\\ \end{align} $$ and the Chebyshev polynomial of the second kind satsifies: $$U_n \left( -\cos \left( \pi \,\beta \right)\right)={\frac { \left( -1 \right) ^{n}\sin \left( \left( 1+n \right) \pi \, \beta \right) }{\sin \left( \pi \,\beta \right) }} \tag{3} $$ so after using $(2,3)$ in $(1)$ and switching integration and summation order we obtain a Fourier series: $$ \begin{align} f(\alpha,\beta)&=\frac{1}{\sin \left( \pi \,\beta \right)} \sum _{n=0}^{\infty }\left( -1 \right) ^{n}\sin \left( \left( 1+n \right) \pi \, \beta \right) \left( \dfrac{1}{1+n+\alpha }+ \dfrac{1}{1+n-\alpha } \right)\tag{4}\\ &=-\frac{1}{\sin \left( \pi \,\beta \right)} \sum _{n=-\infty\,(n\ne0)}^{\infty } \dfrac{\left( -1 \right) ^{n}\sin \left( n \pi \, \beta \right)}{n+\alpha } \\ &=-\frac{1}{\sin \left( \pi \,\beta \right)} \mathfrak{I}\left[ \sum _{n=1}^{\infty }\left( -1 \right) ^{n}e^{i n\pi \beta } \left( \dfrac{1}{n+\alpha }+ \dfrac{1}{n-\alpha } \right)\right]\\ &=-\frac{1}{\sin \left( \pi \,\beta \right)} \mathfrak{I}\left[\Phi(-e^{i \pi \beta },1,\alpha)+\Phi(-e^{i \pi \beta },1,-\alpha)\right]\\ &=-\frac{1}{\sin \left( \pi \,\beta \right)} \mathfrak{I}\left[\Phi_{+}(-e^{i \pi \beta },1,\alpha)\right]\quad:\quad(n=-\infty..+\infty,n\ne 0)\\ &=-\frac{1}{\sin \left( \pi \,\beta \right)} \mathfrak{I}\left[L\left(\frac{\beta+1}{2},\alpha,1\right)+L\left(\frac{\beta+1}{2},-\alpha,1\right)\right]\\ &=-\frac{1}{\sin \left( \pi \,\beta \right)} \mathfrak{I}\left[L_{+}\left(\frac{\beta+1}{2},\alpha,1\right)\right]\\ &=-\frac{1}{2\sin \left( \pi \,\beta \right)}\left[L_{+}\left(\frac{\beta+1}{2},\alpha,1\right)-L_{+}\left(\frac{\beta+1}{2},-\alpha,1\right)\right] \end{align}$$ where $L$ is the Lerch zeta function and $\Phi$ the Lerch transcendant. I can't believe this Fourier series is invariant under $\alpha \leftrightarrow \beta$. I have not as of yet found that symmetry for the Lerch zeta function on line. This series together with the previous demonstration that: $$f(\alpha, \beta) = \pi \frac{ \sin(\pi\alpha \beta)}{\sin(\pi \alpha) \sin(\pi \beta)} \tag{5}$$ show immediately that the following special cases hold which are interesting in their own right: $$\sum _{n=-\infty }^{\infty }{\frac { \left( -1 \right) ^{n}\sin \left( \pi \,xn \right) }{x-n}}={\frac {\pi\sin \left( \pi{x}^{ 2} \right) }{\sin \left( \pi x \right) }}\tag{6}$$ $$\sum _{n=-\infty }^{\infty }{\frac { \left( -1 \right) ^{n}\cos \left( \pi \,xn \right) }{1- \left( n+x \right) ^{2}}}=\pi\sin \left( \pi {x}^{2} \right)\tag{7}$$

Now, from $(4)$ and $(5)$ and the variable change $\alpha=x,\, \beta=2y-1$, with $-1<x<1,\,0\le y<1$, we have: $$L_{+}(y,x,1)=-L_{+}(y,-x,1)-2\pi i\dfrac{\sin(2\pi x(y-\frac{1}{2})}{\sin(x)} \tag{8}$$ where we recognise the trigonometric term as the Dirichlet Kernel (with $x\rightarrow2x$ and for $y$ generalised to non-integer). If we then use differentiation with respect to $x$ as a raising operator we obtain the reflection formula in the $x$ variable: $$L_{+}(y,x,k)=\left( -1 \right) ^{k-1}L_{+}(y,-x,k) -2\pi i \dfrac{ \left( -1 \right) ^{k-1}}{(k-1)!}{\frac {\partial ^{k-1}}{\partial {x}^{k-1}}} {\frac {\sin \left( 2\pi x \left( y-\frac{1}{2} \right) \right) } {\sin \left( \pi x \right) }} \tag{9}$$ where the order becomes $k$ from simple differentiation of the function definition, and it also follows from reversing summation order in the function definition that: $$L_{+}(y,x,k)=\sum_{n=-\infty (n\ne0)}^{\infty}\dfrac{e^{2\pi in y}}{(n+x)^k}=(-1)^k L_{+}(-y,-x,k)\tag{10}$$ and so $(9)$ can also be viewed as a reflection formula in $y$: $$L_{+}(y,x,k)=L_{+}(-y,x,k) -2\pi i \dfrac{ \left( -1 \right) ^{k-1}}{(k-1)!}{\frac {\partial ^{k-1}}{\partial {x}^{k-1}}} {\frac {\sin \left( 2\pi x \left( y-\frac{1}{2} \right) \right) } {\sin \left( \pi x \right) }} \tag{11}$$ or as an explicit formula for the imaginary part: $$\mathfrak{I}\left(L_{+}(y,x,k)\right)= \pi\dfrac{ \left( -1 \right) ^{k-1}}{(k-1)!}{\frac {\partial ^{k-1}}{\partial {x}^{k-1}}} {\frac {\sin \left( 2\pi x \left( y-\frac{1}{2} \right) \right) } {\sin \left( \pi x \right) }} \tag{11}$$ As a final application, evaluating $(9)$ and $(10)$ at $x=0$ and recognising that $(10)$ is then proportional to the Fourier series of the Bernoulli polynomials $B(m,y)$, we obtain the Taylor series for the Dirichlet kernel in the $x$ variable: $$2\sum _{m=0}^{\infty }\left( -1 \right) ^{m}{\frac { B \left( 2m+1,y \right) }{ \left( 2m+1 \right) !}}{x}^{2m}={ \frac {\sin \left( x \left( y-\frac{1}{2} \right) \right) }{\sin \left( \frac{x}{2} \right) }} \tag{12}$$

Still no closer to displaying the symmetry as an integral but I just wanted to show some interesting consequences of the symmetry and the relation itself. Also, if we wanted to preserve the symmetry and generalise equation $(1)$ then we could do so by applying any symmetric differential operator as a raising operator e.g. ${\partial_{\alpha}}{\partial_{\beta}}$.

Not an answer, but a reply to Maesumi's comment --

Invariance under $\alpha \to -\alpha$ is not so difficult to see: let $x=y^{-1}$, then \begin{align*}\int_0^{\infty} \frac{x^{\alpha}}{1+2x\cos(\pi \beta) + x^2} dx &= \int_{\infty}^0 \frac{y^{-\alpha} }{1+2y^{-1}\cos(\pi\beta)+y^{-2}}(-y^{-2})dy\\ &=\int_0^{\infty} \frac{y^{-\alpha}}{1+2y\cos(\pi \beta) + y^2} dy. \end{align*}

No idea about $\alpha \leftrightarrow \beta$ though.