Derivative of L2 Norm of Matrix

Solution 1:

Consider the SVD of $\mathbf{A}=\mathbf{U}\mathbf{\Sigma}\mathbf{V}^T$. It follows that $\mathbf{A}^T\mathbf{A}=\mathbf{V}\mathbf{\Sigma}^2\mathbf{V}$.

The matrix norm is thus related to the maximum singular value of $\mathbf{A}$. Let $s_1$ be such value with the corresponding left and right singular vectors $\mathbf{u}_1$ and $\mathbf{v}_1$.

$$ \| \mathbf{A} \|_2 = \sigma_1(\mathbf{A}) = \sqrt{\lambda_1 \left( \mathbf{A}^T\mathbf{A} \right)} $$

We know that $$d\sigma_1 = \mathbf{u}_1 \mathbf{v}_1^T : d\mathbf{A}$$

It follows that $$ \frac{\partial}{\partial \mathbf{A}} \| \mathbf{A} \|_2^2 = 2 \sigma_1 \mathbf{u}_1 \mathbf{v}_1^T $$