Solving $\int_0^{\infty} \sin(ax^2)\sin(b/x^2)\,\mathrm dx$

$$\int_0^{\infty} \sin ax^2\sin bx^{-2}\,dx=\frac{1}{2}\int_0^{\infty} \cos \left(ax^2-\frac{b}{x^2}\right)\,dx-\frac{1}{2}\int_0^{\infty} \cos \left(ax^2+\frac{b}{x^2}\right)\,dx$$

In general:

$$\begin{aligned} \int_{0}^{\infty}f \left(p x-\frac{q}{x}\right)^2\,dx &= \sqrt{\frac{q}{p}}\int_{0}^{\infty }f \left(\sqrt{pq} x- \frac{\sqrt{pq}}{x}\right)^2\,dx \quad(\textstyle x\to \sqrt{\frac{q}{p}}x )\\ &=\sqrt{\frac{q}{p}}\left[ \int_{0}^{1} f \left(\sqrt{pq} x-\frac{\sqrt{pq}}{x} \right)^2\,dx+ \int_{1}^{\infty} f \left(\sqrt{pq}x-\frac{\sqrt{pq}}{x}\right)^2\,dx \right]\\&= \sqrt{\frac{q}{p}}\left[\int_{1}^{\infty} \left(1+\frac{1}{x^2}\right)f \left(\sqrt{pq} x-\frac{\sqrt{pq}}{x} \right)^2\,dx \right]\quad( x\to x^{-1})\\&=\frac{1}{p} \int_{0}^{\infty} f (t^2 )\,dt \quad \big(t=\sqrt{pq} x-\sqrt{pq}x^{-1}\big) \end{aligned}$$

Observe that:

$$\begin{aligned}\int_0^{\infty} \cos \left(ax^2+\frac{b}{x^2}\right)\,dx&=\cos 2\sqrt{ab}\int_{0}^{\infty} \cos \left(\sqrt{a}x-\frac{\sqrt{b}}{x}\right)^2\,dx\\&\;-\sin 2\sqrt{ab}\int_{0}^{\infty} \sin \left(\sqrt{a}x-\frac{\sqrt{b}}{x}\right)^2\,dx\end{aligned}$$

Now since $\displaystyle \int_0^{\infty}\cos x^2\,dx=\int_0^{\infty}\sin x^2\,dx=\sqrt{\frac{\pi}{8}}$ using our previous result we have:

$$\int_0^{\infty} \cos \left(ax^2+\frac{b}{x^2}\right)\,dx=\sqrt{\frac{\pi}{8a}}\left(\cos 2\sqrt{ab}-\sin 2\sqrt{ab}\right)$$

Similarly, writing:

$$\begin{aligned}\int_0^{\infty} \cos \left(ax^2-\frac{b}{x^2}\right)\,dx&=\cosh 2\sqrt{ab}\int_{0}^{\infty} \cos \left(\sqrt{a}x-i\frac{\sqrt{b}}{x}\right)^2\,dx\\&\;-\sinh 2\sqrt{ab}\int_{0}^{\infty} \sin \left(\sqrt{a}x-i\frac{\sqrt{b}}{x}\right)^2\,dx\end{aligned}$$

We get:

$$\int_0^{\infty} \cos \left(ax^2-\frac{b}{x^2}\right)\,dx=\sqrt{\frac{\pi}{8a}}\left(\cosh 2\sqrt{ab}-\sinh 2\sqrt{ab}\right)$$

Putting everything together:

$$\int_0^{\infty} \sin ax^2\sin bx^{-2}\,dx=\frac{1}{2}\sqrt{\frac{\pi}{8a}}\Bigg\{\big(\cosh 2\sqrt{ab}-\sinh 2\sqrt{ab}\big)-\big(\cos 2\sqrt{ab}-\sin 2\sqrt{ab}\big)\Bigg\}$$

Consider

$$\int_0^{\infty} dx \, e^{i (a x^2 - b/x^2)}$$

By considering a wedge in the complex plane of angle $\pi/4$ and using Cauchy's theorem, you can show that this integral is equal to

$$e^{i \pi/4} e^{2 \sqrt{a b}} \int_0^{\infty} dx \, e^{-(\sqrt{a} x + \sqrt{b}/x)^2}$$

Let $u = \sqrt{a} x + \sqrt{b}/x$, then

$$dx = \frac{1}{2 \sqrt{a}}\left (1 \pm \frac{u}{\sqrt{u^2-4 \sqrt{a b}}} \right )du$$

Note there are two branches, so that integration over $u$ results in two integrals:

$$e^{i \pi/4} \frac{e^{2 \sqrt{a b}}}{2 \sqrt{a}} \left [\int_{2 \sqrt{\sqrt{a b}}}^{\infty} du \left (1 + \frac{u}{\sqrt{u^2-4 \sqrt{a b}}} \right ) e^{-u^2} - \int_{2 \sqrt{\sqrt{a b}}}^{\infty} du \left (1 - \frac{u}{\sqrt{u^2-4 \sqrt{a b}}} \right ) e^{-u^2}\right ] $$

which simplifies to

$$e^{i \pi/4} \frac{e^{2 \sqrt{a b}}}{\sqrt{a}}\int_{2 \sqrt{\sqrt{a b}}}^{\infty} du \frac{u}{\sqrt{u^2-4 \sqrt{a b}}} e^{-u^2}$$

which turns out to be very integrable; the result is

$$\int_0^{\infty} dx \, e^{i (a x^2 + b/x^2)}=e^{i \pi/4} \sqrt{\frac{\pi}{a}}e^{-2 \sqrt{a b}}$$

Now consider

$$\int_0^{\infty} dx \, e^{i (a x^2 + b/x^2)}$$

By the same reasoning, this integral is equal to

$$e^{i \pi/4} \sqrt{\frac{\pi}{a}}e^{-i 2 \sqrt{a b}}$$

The desired integral is one-half the real part of the difference between these two integrals, i.e.,

$$\int_0^{\infty} dx \sin{\left ( a x^2\right )} \sin{\left (\frac{b}{x^2}\right)} = \frac12 \Re{\left[\int_0^{\infty} dx \, e^{i (a x^2 - b/x^2)} - \int_0^{\infty} dx \, e^{i (a x^2 + b/x^2)} \right]}$$

Therefore, converting the exponential to hyperbolic functions (for symmetry), I get

$$\int_0^{\infty} dx \sin{\left ( a x^2\right )} \sin{\left (\frac{b}{x^2}\right)} = \frac14 \sqrt{\frac{\pi}{2 a}} \left[\left (\cosh{\left(2 \sqrt{a b}\right)}-\cos{\left(2 \sqrt{a b}\right)}\right) - \left (\sinh{\left(2 \sqrt{a b}\right)}-\sin{\left(2 \sqrt{a b}\right)}\right) \right]$$

Answer revamped. Here is an answer which is free of heavy computation (with the exception where we utilized the Glasser's master theorem).

If you do not bother with the rigor, it is OK to skip the text and focus on computations.

Let us consider the function

$$ I(a, b) = \int_{0}^{\infty} \exp\left\{ i \left( a x^{2} + \frac{b}{x^2} \right) \right\} \, dx. \tag{*} $$

Denote by $\mathbb{H} = \{z \in \mathbb{C} : \operatorname{Im}(z) > 0 \}$ the upper half-plane and treat $a$ and $b$ as variable on the the closed upper-half plane $\bar{\mathbb{H}}$. Then $I$ defines a holomorphic function on $\mathbb{H}\times\mathbb{H}$. Moreover, a form of Abelian theorem tells that for any $a, b \in \mathbb{R}\setminus\{0\}$ we have

$$ I(a, b) = \lim_{h_1 \to 0^+} \lim_{h_2 \to 0^+} I(a+ih_1, b + ih_2). \tag{1} $$

(The order of the limit above is not so important. So let us just stick to this particular order.) In view of these properties, together with the principle of analytic continuation, it suffices to identify $I(a, b)$ when $a, b$ lie on the positive imaginary axis. So let $a, b > 0$. Then by the Glasser's master theorem,

\begin{align*} I(ia, ib) &= \int_{0}^{\infty} \exp\bigg\{ -\left( ax^2 + \frac{b}{x^2} \right) \bigg\} \, dx \\ &= \frac{1}{2}e^{-2\sqrt{ab}} \int_{-\infty}^{\infty} \exp\bigg\{ - a \bigg( x - \frac{\sqrt{b/a}}{x} \bigg)^2 \bigg\} \, dx \\ &= \frac{1}{2}e^{-2\sqrt{ab}} \int_{-\infty}^{\infty} e^{-ax^2} \, dx \qquad \text{($\because$ Glasser)} \\ &= \frac{\sqrt{\pi}}{2\sqrt{a}} e^{-2\sqrt{ab}} = \frac{\sqrt{i\pi}}{2\sqrt{ia}} e^{2i\sqrt{ia}\sqrt{ib}}. \tag{2} \end{align*}

Since the principal square root $z \mapsto \sqrt{z}$ defines a holomorphic function on $\mathbb{H}$, the above computation extends to all of $\mathbb{H}\times\mathbb{H}$, yielding

$$ I(a, b) \stackrel{(2)}{=} \frac{\sqrt{i\pi}}{2\sqrt{a}} e^{2i\sqrt{a}\sqrt{b}}. \tag{3} $$

Then $\text{(1)}$ tells that the result above extends to all of $a, b \in \mathbb{R}\setminus\{0\}$, where the branch for the principal logarithm is chosen to be $-\pi < \arg(z) \leq \pi$. Then applying the relation

$$ \begin{split} \sin A \sin B & = \frac{e^{iA} - e^{-A}}{2i} \times \frac{e^{iB} - e^{-iB}}{2i} \\ &= \frac{1}{4}(e^{i(A-B)} + e^{i(B-A)} - e^{i(A+B)} - e^{-i(A+B)}), \end{split} \tag{4} $$

to the identity $\text{(3)}$ and using the relation $\sqrt{-x} = i\sqrt{x}$ for $x > 0$, we obtain

\begin{align*} \int_{0}^{\infty} \sin(ax^2)\sin\left(\frac{b}{x^2}\right) \, dx &\stackrel{(4)}{=} \frac{1}{4}(I(a,-b) + I(-a,b) - I(a,b) - I(-a,-b)) \\ &\stackrel{(3)}{=} \frac{(1+i)\sqrt{\pi}}{8\sqrt{2a}}\left( e^{-2\sqrt{ab}} - i e^{-2\sqrt{ab}} - e^{2i\sqrt{ab}} + i e^{-2i\sqrt{ab}} \right) \\ &= \frac{\sqrt{\pi}}{4\sqrt{2a}}\left( e^{-2\sqrt{ab}} - \cos(2\sqrt{ab}) + \sin(2\sqrt{ab}) \right). \end{align*}