Limit definition of pseudoinverse: $A^+ b$ is as close as possible to $y$ in terms of the Euclidean norm $\lVert Ax-b\rVert_2$

Solution 1:

We assume that $(A^TA + \alpha I)^{-1}A^T$ indeed has a limit as $\alpha \to 0^+$. Let $x$ be given by $$ x = A^+ y = \lim_{\alpha \to 0^+ }[(A^TA + \alpha I)^{-1}A^T y]. $$

Consider any $\alpha > 0$. We note that $x_\alpha = (A^TA + \alpha I)^{-1}A^Ty$ is the unique solution to the system $$ (A^TA + \alpha I)x_{\alpha} = A^Ty, $$ and that $x_{\alpha} \to x$ as $\alpha \to 0^+$. It follows that $$ \|A^Ty - A^TAx\| = \lim_{\alpha \to 0^+}\|A^T y - A^TAx_{\alpha}\| = \lim_{\alpha \to 0^+}\|\alpha x_{\alpha}\| \\ \qquad \qquad = \lim_{\alpha \to 0^+} \alpha \|x_{\alpha}\| = \lim_{\alpha \to 0^+} \alpha \|x\| = 0. $$ So, we indeed have $A^TAx = A^Ty$, which means that $x$ is a least-squares solution to $Ax = y$, which is what we wanted.