Integral: $\int_0^{\infty} \cos\left(\frac{a^2}{x^2}-b^2x^2\right)\,dx$ for $a,b>0$

The confusion arises from the fact that you're assuming the formula $$\int_0^{\infty} e^{-\left(p^2x^2+m^2/x^2\right)}\,dx=\frac{\sqrt{\pi}}{2p}e^{-2pm} $$ is valid if $p^{2}$ is imaginary and positive and $m^{2}$ is imaginary and negative. But that formula is usually derived under the condition that $p$ and $m$ are real positive parameters.

But by integrating on the complex plane, you can see how the two integrals are related.

Let $ \displaystyle f(z) = e^{i b^{2}z^{2}} e^{-ia^{2}/z^{2}}$ and integrate around a wedge/sector of radius $R$ that makes an angle of $ \frac{\pi}{4}$ with the positive real axis and is indented around the essential singularity at the origin.

Along the arc of the wedge, $ \displaystyle |e^{-ia^{2}/z^{2}}|= |e^{-i a^{2}/(R^{2}e^{2it})}| =e^{-a^{2} \sin 2t/R^{2}} \le 1 $ since $ \displaystyle 0 \le t \le \frac{\pi}{4}$.

Therefore,

$$\begin{align} \Big| \int_{0}^{\pi /4} f(Re^{it}) \ i Re^{it} \ dt \Big| &\le R \int_{0}^{\pi /4} e^{-b^{2} R^{2} \sin 2t} \ dt \\ &\le R \int_{0}^{\pi /4} e^{-b^{2} R^{2} \frac{4}{\pi} t } \ dt \ \ \text{(Jordan's inequality)} \\ &= \frac{\pi}{4} \frac{1}{b^{2}R} \Big( 1-e^{-b^{2}R^{2}}\Big) \to 0 \ \text{as} \ R \to \infty .\end{align}$$

A very similar argument shows that the integral also vanishes along the quarter-circle indentation around the origin as the radius of the quarter-circle goes to $0$.

Therefore, since $f(z)$ is analytic inside and on the contour,

$$ \int_{0}^{\infty} f(x) \ dx - \int_{0}^{\infty} f(te^{i \pi /4}) e^{i \pi /4} \ dt =0$$

which implies

$$ \begin{align}\int_{0}^{\infty}e^{i b^{2}x^{2}} e^{-ia^{2}/x^{2}} \ dx &= e^{i \pi /4} \int_{0}^{\infty} e^{-b^{2}t^{2}} e^{-a^{2}/t^{2}} \ dt \\ &= e^{i \pi /4} \frac{\sqrt{\pi}}{2b} e^{-2 ab}. \end{align}$$

And equating the real parts on both sides of the equation,

$$ \begin{align} \int_{0}^{\infty} \cos \left(b^{2}x^{2} -\frac{a^{2}}{x^{2}} \right) \ dx &= \int_{0}^{\infty} \cos \left(\frac{a^{2}}{x^{2}} - b^{2} x^{2} \right) \ dx \\ &= \frac{1}{2b} \sqrt{\frac{\pi}{2}} e^{-2 ab} . \end{align}$$

I confirm that, before any simplification, $$I=\int_0^{\infty} e^{i \left(\frac{a^2}{x^2}-b^2 x^2\right)} \, dx=\frac{\sqrt{\pi } e^{-\frac{2 \sqrt{i b^2}}{\sqrt{\frac{i}{a^2}}}}}{2 \sqrt{i b^2}}$$ under the conditions that $\Im\left(b^2\right)<0\land \Im\left(a^2\right)>0$.

After simplifications $$J=\int_0^{\infty} \cos\left(\frac{a^2}{x^2}-b^2x^2\right)\,dx=\Re\left(\frac{1}{2b}\sqrt{\frac{\pi}{i}}e^{-2ab}\right)=\frac{1}{2b}\sqrt{\frac{\pi}{2}}e^{-2ab}$$ which is your answer with a minus sign in the exponent.

I performed numerical checks and this is correct.