Solution 1:

The integral should be from $0\to\infty$. Because the pdf is non-zero only for positive $x$. So you should do that or while writing the integral from $-\infty\to\infty$ you should also include the $I_{(0,\infty)}$ inside the integral.

$$\int_{0}^{\infty} e^{-(\lambda x)^2(-t+1)} 2x\lambda^2 dx$$

Now you substitute $\lambda^{2}x^{2}(1-t)=z$ so

$2\lambda^{2}xdx=\frac{1}{1-t}dz$

So you get:-

$$\int_{0}^{\infty}\frac{1}{1-t}e^{-z}\,dz=\frac{1}{1-t}$$

Which is the MGF of an $\text{Exp}(1)$ distribution.

So $Y\sim \text{Exp}(1)$

PS:- As an advice, when you are asked to do something by a more complicated method, it is better to first do the problems by easier methods with which you are comfortable with and then proceed with the problem. Then you will know that if the answers don't match up, you might have made a mistake .

For example, it is way easier to just use the Jacobian transformation and say that $Y\sim\text{Exp}(1)$.

To do that you have $Y=X^{2}\lambda^{2}$.

So $\frac{dy}{dx}=2x\lambda^{2}$.

So $|\frac{dx}{dy}|=\frac{1}{2x\lambda^{2}}$

So $f_{Y}(y)=|\frac{dx}{dy}|f_{X}(x)=\frac{1}{2x\lambda^{2}}e^{-(\lambda x)^2} 2x\lambda^{2} I_{(0,\infty)}$

So $f_{Y}(y)=e^{-\lambda^{2}x^{2}}I_{(0,\infty)}=e^{-y}I_{(0,\infty)}$

So you see that the pdf of $Y$ has the same pdf of a $\text{Exp}(1)$ distribution. That lets you easily conclude as it directly gives you the pdf rather than checking which MGF it matches with.