tough integral involving $\sin(x^2)$ and $\sinh^2 (x)$

It can be done using contour integration and the calculus of residues.

Sketch: Integrate $$ f(z) = \frac{e^{i\pi z^2} e^{\pi z}}{\sinh^2 (\pi z) \cosh(\pi z)} $$ around a rectangular contour with corners at $\pm R$ and $\pm R + i$ and with semicircular indentations of radius $\epsilon$ to avoid the poles at $0$ and $i$, take imaginary parts and let $R\to\infty$, $\epsilon\to 0^+$.

You'll need to use $$ f(x)-f(x+i)=\frac{2 e^{i \pi x^2}}{\sinh^2(\pi x)} $$ together with $$ \operatorname*{res}_{z=0} \, f(z) = \operatorname*{res}_{z=i} \, f(z) = \frac{1}{\pi} $$ (since these will each contribute $-i \pi$ times the residue in the limit $\epsilon \to 0^+$) and $$ \operatorname*{res}_{z=i/2} \, f(z) = \frac{-1+i}{\pi\sqrt{2}}. $$

Although this question is two years old, the integral was mentioned in chat recently, I evaluated it, and then found this question. Since there is no complete solution, although Hans Lundmark's suggestion is excellent and similar in nature, I am posting what I have done.

Contours

Since the integrand is even, $$ \begin{align} \int_0^\infty\frac{\sin(\pi x^2)}{\sinh^2(\pi x)}\,\mathrm{d}x &=\frac12\int_{-\infty}^\infty\frac{\sin(\pi x^2)}{\sinh^2(\pi x)}\,\mathrm{d}x \end{align} $$ Define $$ f(z)=\frac{\cos\left(\pi z^2\right)}{\sinh(2\pi z)\sinh^2(\pi z)} $$ Note that because $$ f(x\pm i) =\frac{-\cos\left(\pi x^2\right)\cosh(2\pi x)\pm i\sin\left(\pi x^2\right)\sinh(2\pi x)}{\sinh(2\pi x)\sinh^2(\pi x)}\\ $$ we have $$ \begin{align} \int_\gamma f(z)\,\mathrm{d}z &=\int_{-\infty}^\infty\big[f(x-i)-f(x+i)\big]\,\mathrm{d}x\\ &=-2i\int_{-\infty}^\infty\frac{\sin(\pi x^2)}{\sinh^2(\pi x)}\,\mathrm{d}x\\ &=2\pi i\times\begin{array}{}\text{the sum of the residues}\\\text{inside the contour}\end{array} \end{align} $$ where $\gamma$ is the contour

$\hspace{3.2cm}$

Therefore, $$ \int_0^\infty\frac{\sin(\pi x^2)}{\sinh^2(\pi x)}\,\mathrm{d}x =-\frac\pi2\times\begin{array}{}\text{the sum of the residues}\\\text{inside the contour}\end{array} $$ Residues

near $0$ : $$ \begin{align} f(z) &=\frac{\cos\left(\pi z^2\right)}{\sinh(2\pi z)\sinh^2(\pi z)}\\ &=\frac{1-\frac12\pi^2z^4+O(z^8)}{2\pi z\left(1+\frac23\pi^2z^2+O(z^4)\right)\pi^2 z^2\left(1+\frac13\pi^2z^2+O(z^4)\right)}\\ &=\frac{1-\pi^2z^2}{2\pi^3z^3}+O(z)\\[10pt] &\implies\text{residue}=-\frac1{2\pi} \end{align} $$ at $\pm i/2$, use L'Hosptal : $$ \begin{align} \text{residue} &=\lim_{z\to\pm i/2}\frac{(z\mp i/2)\cos\left(\pi z^2\right)}{\sinh(2\pi z)\sinh^2(\pi z)}\\ &=\frac1{2\pi\cosh(\pm\pi i)}\frac{\cos(-\pi/4)}{\sinh^2(\pm\pi i/2)}\\ &=\frac1{2\pi\cos(\pm\pi)}\frac{\sqrt2/2}{-\sin^2(\pm\pi/2)}\\[4pt] &=\frac{\sqrt2}{4\pi} \end{align} $$ near $\pm i$ : $$ \begin{align} f(z\pm i) &=\frac{-\cos\left(\pi z^2\right)\cosh(2\pi z)\pm i\sin\left(\pi z^2\right)\sinh(2\pi z)}{\sinh(2\pi z)\sinh^2(\pi z)}\\ &=\frac{-\left(1-\frac12\pi^2z^4+O(z^8)\right)\left(1+2\pi^2z^2+O(z^4)\right)+O(z^3)}{2\pi z\left(1+\frac23\pi^2z^2+O(z^4)\right)\pi^2 z^2\left(1+\frac13\pi^2z^2+O(z^4)\right)}\\ &=-\frac{1+\pi^2z^2}{2\pi^3z^3}+O(1)\\[10pt] &\implies\text{residue}=-\frac1{2\pi} \end{align} $$ Result

Thus, $$ \begin{align} \int_0^\infty\frac{\sin(\pi x^2)}{\sinh^2(\pi x)}\,\mathrm{d}x &=-\frac\pi2\left(-\frac1{2\pi}-\frac1{2\pi}+\frac{\sqrt2}{4\pi}+\frac{\sqrt2}{4\pi}\right)\\[6pt] &=\frac{2-\sqrt2}{4} \end{align} $$

I would write the $\sin(x^2)$ as $(e^{ix^2}-e^{-ix^2})/2i$ and the sinh as $(e^{ x}-e^{-x})/2$. Then I'd maybe put the integrand in the form of $(e^{p_1(x)}+e^{p_2(x)}+\cdots)^{-1}+(e^{p_3(x)}+e^{p_4(x)}+\cdots)^{-1}+\cdots$ where $p_i(x)$ are polynomes with complex coefficients. I have no clue if that helps, to be honest.

Another idea would be partial integration after multiplying with 1, like: $\int\mathrm dx 1\cdot f(x)= xf(x)-\int\mathrm dx \; x\cdot f'(x)$ Sometimes this helps to handle a $x^2$ in the argument of a complicated function.