Evaluate $\int_0^\infty \frac{(\mathrm{e}^{-ax}-\mathrm{e}^{-bx})^2}{x^2}dx=\ln\frac{(2a)^{2a}(2b)^{2b}}{(a+b)^{2(a+b)}} $

Write $\frac{1}{x^2}=\int_0^\infty ye^{-xy}dy$ so the LHS of (1) is$$\begin{align}&\int_0^\infty dy\:y\int_0^\infty dx(e^{-(2a+y)x}-2e^{-(a+b+y)x}+e^{-(2b+y)x})\\&=\int_0^\infty dy\:y\left(\frac{1}{2a+y}-\frac{2}{a+b+y}+\frac{1}{2b+y}\right).\end{align}$$You can handle (2) the same way, thought it helps to first replace $\int_0^\infty$ with $\tfrac12\int_{-\infty}^\infty$, so you just have to evaluate Gaussians on $\Bbb R$.

As fjaclot says, $(2)$ is wrong. Making the substitution $u = x\sqrt{a}$ yields $$\sqrt{a}\int_{0}^{\infty}\frac{e^{-u^{2}}-e^{-tu^{2}}}{u^{2}}du$$ where $t = \frac{b}{a}$. Let $f(t)$ be the integral. Then $$f'(t) = \int_0^{\infty} e^{-tu^2} du = \frac{\sqrt{\pi}}{2\sqrt{t}}$$

Since $f(1) = 0$, $(2)$ is $$\sqrt{a}\left( \sqrt{\pi t}-\sqrt{\pi} \right) = \sqrt{\pi}\left(\sqrt{b}-\sqrt{a}\right)$$

Similarly for $(1)$, make the substitution $u=ax$ and let $f(t) = \int_{0}^{\infty}\frac{\left(e^{-u}-e^{-tu}\right)^{2}}{u^{2}}du$. Then $$f'(t) = 2\int_{0}^{\infty}\frac{e^{-\left(1+t\right)u}-e^{-2tu}}{u}du$$

Then you can apply that formula you found so that $$f'(t) = 2 \ln\left( \frac{2t}{1+t} \right) = 2\ln\left(2t\right)-2\ln\left(1+t\right)$$

The integral is then $f(t) = 2t\ln\left( \frac{2t}{t+1} \right)-2\ln(t+1)+2\ln(2)$. Therefore the answer for $(1)$ is $af\left(\frac{b}{a}\right)$ which simplifies to $$\ln\left(\frac{(2a)^{2a}(2b)^{2b}}{(a+b)^{2(a+b)}}\right)$$

Integrate (1) by parts with $\frac{dx}{x^2}=-d(\frac1x)$, \begin{align} &\int_0^\infty \frac{(\mathrm{e}^{-ax}-\mathrm{e}^{-bx})^2}{x^2}dx\\ = &2a \int_0^\infty \frac{{e}^{-(a+b)x}-{e}^{-2ax}}{x}dx + 2b \int_0^\infty \frac{{e}^{-(a+b)x}-{e}^{-2bx}}{x}dx\\ = &2a \ln \frac{2a}{a+b} + 2b \ln \frac{2b}{a+b} =\ln\frac{(2a)^{2a}(2b)^{2b}}{(a+b)^{2(a+b)}} \end{align} where $ \int_0^\infty \frac{\mathrm{e}^{-px }-\mathrm{e}^{-qx }}{x}\mathrm{d}x=\ln\frac{q}{p} $ is used.

Integrate (2) by parts $$\int_0^\infty \frac{{e}^{-ax^2}-{e}^{-bx^2}}{x^2}{d}x= 2\int_0^\infty (b{e}^{-bx^2}-a{e}^{-ax^2})dx= \sqrt{\pi}(\sqrt b-\sqrt a) $$ where $\int_0^\infty e^{-px^2} =\frac{\sqrt{\pi}}{2\sqrt p}$ is used