Contour Integration and Branch Cuts

Consider $f(\omega) = \omega^{-3/4} (x-\omega)^{-3/4} (1-\omega)^{-3/4}$ and $ I(x) = \oint_{\vert \omega \vert = x} f(\omega) \mathrm{d} \omega$.

The function $f(\omega)$ is discontinuous at $\omega = -x$ along $\vert \omega \vert = x$ and has integrable singularity at $\omega = x$. Making a change of variables, $\omega = x z$: $$ I(x) = \frac{1}{\sqrt{x}} \oint_{\vert z \vert = 1} z^{-3/4} (1-z)^{-3/4} (1-x z)^{-3/4} \mathrm{d} z = \frac{1}{\sqrt{x}} \oint_{\vert z \vert = 1} h\left(z\right) \, \mathrm{d} z $$

Let $\mathcal{C}$ denote the circle $\vert z \vert = 1$. Let $C_{-1, \delta}$ denote segment of $\mathcal{C}$ which crosses the negative axis and of length $2 \pi \delta$, with $\pi \delta$ above and $\pi \delta$ below the negative axis.

It is clear that integral along $\mathcal{C}_{-1,\delta}$ is vanishing as $\delta \to 0$: $$ \begin{eqnarray} \left\vert \int_{\mathcal{C}_{-1,\delta}} z^{-3/4} (1-z)^{-3/4} (1-x z)^{-3/4} \mathrm{d} z \right\vert &\le& (2(1+x))^{-3/4} \left\vert \int_{\mathcal{C}_{-1,\delta}} z^{-3/4} \mathrm{d} z \right\vert \\ &=& \left(2 (1+x) \right)^{-3/4} \frac{8\sqrt{2}}{7} \sin\left(\frac{7 \delta}{8}\right) \left( \cos\left(\frac{7 \delta}{8}\right) - \sin\left(\frac{7 \delta}{8}\right) \right) \\ &\le& \sqrt{2} \delta \left(2 (1+x) \right)^{-3/4} \end{eqnarray} $$

Let's complete $\mathcal{C} \backslash \mathcal{C}_{-1,\delta}$ with integration along $(-1,0)$ above the axis and then along $(0,-1)$ below the axis so as to complete the contour, and call the completed contour $\mathcal{L}$. Then $$ \begin{eqnarray} \oint_{\mathcal{C}} h\left(z\right) \, \mathrm{d} z &=& \oint_{\mathcal{L}} h\left(z\right) \, \mathrm{d} z + \int_0^1 \left( h\left(-y - i \epsilon \right) - h\left(-y + i \epsilon \right) \right)\, \mathrm{d} y \\ &=& \oint_{\mathcal{L}} h\left(z\right) \, \mathrm{d} z + \left( \mathrm{e}^{i \frac{3 \pi}{4}} - \mathrm{e}^{-i \frac{3 \pi}{4}} \right) \int_0^1 y^{-3/4}(1+y)^{-3/4}(1+x y)^{-3/4} \mathrm{d} y \end{eqnarray} $$ The claim is that the principal value of $\oint_\mathcal{L} h(z) \mathrm{d} z = 0$, so we get $$ I(x) = \frac{2 i \sin\left( 3/4 \pi \right)}{\sqrt{x}} \int_0^1 y^{-3/4}(1+y)^{-3/4}(1+x y)^{-3/4} \mathrm{d} y $$

Now, let's check this with quadratures:

Notice that the purported answer you gave in your post can not be correct, as it is not purely imaginary.

I think the authors get the answer just by contour deformation from the circle to the green line. Suppose above the branch cut $(0,x)$, the integrated function is $$(1-\omega)^{-3/4}(x-\omega)^{-3/4}\omega^{-3/4}$$ When the contour rotates around $\omega=0$, we have $$\omega\rightarrow \omega e^{2\pi i}$$ and the integrated function becomes $$e^{-3\pi i/2}(1-\omega)^{-3/4}(x-\omega)^{-3/4}\omega^{-3/4}$$