Inverse of symmetric positive definite perturbation of symmetric positive definite matrix

Solution 1:

For my convenience, I will replace $\delta A$ with $H$. Note that $$ (A + H)^{-1} = (A^{1/2}[I + A^{-1/2}HA^{-1/2}]A^{1/2}) \\ = A^{-1/2}(I + A^{-1/2}HA^{-1/2})A^{-1/2}, $$ so that $$ (A + H)^{-1} - A^{-1} = \\ A^{-1/2}[(I + K)^{-1} - I]A^{-1/2}, $$ where $K = A^{-1/2}HA^{-1/2}$. It follows that $$ \frac{\|(A + H)^{-1} - A^{-1}\|}{\|A^{-1}\|} = \frac{\|A^{-1/2}[(I + K)^{-1} - I]A^{-1/2}\|}{\|A^{-1}\|} \\ \leq \frac{\|A^{-1/2}\|\cdot \|(I + K)^{-1} - I\| \cdot \|A^{-1/2}\|}{\|A^{-1}\|} \\ = \frac{\|A^{-1}\|\cdot \|(I + K)^{-1} - I\|}{\|A^{-1}\|} \\ = \|(I + K)^{-1} - I\|. $$ Now, $K = (I + K) - I$ is positive definite. It follows that $I^{-1} - (I + K)^{-1} = K(I + K)^{-1}$ is positive definite. Thus, noting that $x \mapsto x/(1 + x)$ is increasing for $x \geq 0$, we have $$ \|(I + K)^{-1} - I\| = \lambda_{\max}(K(I + K)^{-1}) = \lambda_{\max}(K)(1 + \lambda_{\max}(K))^{-1}. $$ Notably, we have $$ \lambda_{\max}(K) = \|A^{-1/2}HA^{-1/2}\| \leq \|A^{-1/2}\|^2 \|H\| = \|A^{-1}\| \cdot \|H\|. $$ All together, this gets us the inequality $$ \frac{(A + H)^{-1} - A^{-1}}{\|A^{-1}\|} \leq \frac{\|A^{-1}\|\cdot \|H\|}{1 + \|A^{-1}\| \cdot \|H\|} < \|A^{-1}\| \cdot \|H\| = \kappa(A) \frac{\|H\|}{\|A\|}. $$