Visualization of Singular Value decomposition of a Symmetric Matrix

Singular value decomposition

Start with a matrix with $m$ rows, $n$ columns, and rank $\rho$, $$ \mathbf{A}\in\mathbb{C}^{m\times n}_{\rho} $$ which has the singular value decomposition $$ \mathbf{A} = \mathbf{U} \, \Sigma \, \mathbf{V}^{*} = % \left[ \begin{array}{cc} \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} & \color{red} {\mathbf{U}_{\mathcal{N}\left(\mathbf{A}^{*}\right)}} \end{array} \right] % \left[ \begin{array}{cc} \mathbf{S} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right] % \left[ \begin{array}{cc} \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} & \color{red} {\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}} \end{array} \right]^{*} % $$ where the color denotes $\color{blue}{range}$ spaces and $\color{red}{null}$ spaces. The dimensions of the domain matrices are $$ % \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} \in \mathbb{C}^{m\times \rho}, \quad % \color{red}{\mathbf{U}_{\mathcal{N}\left(\mathbf{A}^{*}\right)}} \in \mathbb{C}^{m \times m - \rho}, \quad % \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} \in \mathbb{C}^{n\times \rho}, \quad % \color{red}{\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}} \in \mathbb{C}^{n\times n - \rho}. $$ The domain matrices are unitary: $$ \begin{align} \mathbf{U}\mathbf{U}^{*} &= \mathbf{U}^{*}\mathbf{U} = \mathbf{I}_{m} \\ \mathbf{V}\mathbf{V}^{*} &= \mathbf{V}^{*}\mathbf{V} = \mathbf{I}_{n} \end{align} $$

The dimensions of the singular value matrices are $$ % \Sigma \in \mathbb{R}^{m\times n}, \quad % \mathbf{S} \in \mathbb{R}^{\rho\times \rho}. $$

The hermitian conjugate is constructed according to $$ \mathbf{A}^{*} = \mathbf{V} \, \Sigma^{\mathrm{T}} \, \mathbf{U}^{*} = % \left[ \begin{array}{cc} \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} & \color{red} {\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}} \end{array} \right] % \left[ \begin{array}{cc} \mathbf{S} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right] % \left[ \begin{array}{cc} \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} & \color{red} {\mathbf{U}_{\mathcal{N}\left(\mathbf{A}^{*}\right)}} \end{array} \right]^{*} % $$ where $\Sigma^{\mathrm{T}}\in \mathbb{R}^{n\times m}$.

The Moore-Penrose pseudoinverse is constructed according to $$ \mathbf{A}^{\dagger} = \mathbf{V} \, \Sigma^{\dagger} \, \mathbf{U}^{*} = % \left[ \begin{array}{cc} \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} & \color{red} {\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}} \end{array} \right] % \left[ \begin{array}{cc} \mathbf{S}^{-1} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right] % \left[ \begin{array}{cc} \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} & \color{red} {\mathbf{U}_{\mathcal{N}\left(\mathbf{A}^{*}\right)}} \end{array} \right]^{*} % $$ where $\Sigma^{\dagger}\in \mathbb{R}^{n\times m}$.

The product matrix rules you stated always hold: $$ \begin{align} % \mathbf{A} \mathbf{A}^{*} &= % \left( \mathbf{U} \, \mathbf{\Sigma} \, \mathbf{V}^{*} \right) % \left( \mathbf{U} \, \mathbf{\Sigma} \, \mathbf{V}^{*} \right)^{*} % = % \left( \mathbf{U} \, \mathbf{\Sigma} \, \mathbf{V}^{*} \right) % \left( \mathbf{V} \, \mathbf{\Sigma}^{\mathrm{T}} \, \mathbf{V}^{*} \right) \\ % \mathbf{A}^{*} \mathbf{A} &= % \left( \mathbf{U} \, \mathbf{\Sigma} \, \mathbf{V}^{*} \right)^{*} % \left( \mathbf{U} \, \mathbf{\Sigma} \, \mathbf{V}^{*} \right) % = % \left( \mathbf{V} \, \mathbf{\Sigma}^{\mathrm{T}} \, \mathbf{V}^{*} \right) % \left( \mathbf{U} \, \mathbf{\Sigma} \, \mathbf{V}^{*} \right) \\ % \end{align} $$ Examples follow.

Square, full rank $m = n = \rho$

$$ \mathbf{A} = \mathbf{U} \, \Sigma \, \mathbf{V}^{*} = % \left[ \begin{array}{c} \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} \end{array} \right] % \left[ \mathbf{S} \right] % \left[ \begin{array}{c} \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} \end{array} \right]^{*} % $$ The product matrices are $$ \begin{align} % \mathbf{A}^{*}\mathbf{A} &= \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} \, \mathbf{S}^{2} \, \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}}^{*} \\ % \mathbf{A}\mathbf{A}^{*} &= \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} \, \mathbf{S}^{2} \, \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}}^{*} % \end{align} $$

Tall, full column rank $n = \rho$, $m \ge n$

$$ \mathbf{A} = \mathbf{U} \, \Sigma \, \mathbf{V}^{*} = % \left[ \begin{array}{cc} \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} & \color{red} {\mathbf{U}_{\mathcal{N}\left(\mathbf{A}^{*}\right)}} \end{array} \right] % \left[ \begin{array}{c} \mathbf{S} \\ \mathbf{0} \end{array} \right] % \left[ \begin{array}{cc} \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} & \color{red} {\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}} \end{array} \right]^{*} % $$ The product matrices are $$ \begin{align} % \mathbf{A}^{*}\mathbf{A} &= \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} \, \mathbf{S}^{2} \, \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}}^{*} \\ % \mathbf{A}\mathbf{A}^{*} &= % \left[ \begin{array}{cc} \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} & \color{red} {\mathbf{U}_{\mathcal{N}\left(\mathbf{A}^{*}\right)}} \end{array} \right] % \left[ \begin{array}{cc} \mathbf{S}^{2} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right] % \left[ \begin{array}{cc} \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} & \color{red} {\mathbf{U}_{\mathcal{N}\left(\mathbf{A}^{*}\right)}} \end{array} \right]^{*} % \end{align} $$

Wide, full row rank $m = \rho$, $n \ge m$

$$ \mathbf{A} = \mathbf{U} \, \Sigma \, \mathbf{V}^{*} = % \left[ \begin{array}{c} \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} \end{array} \right] % \left[ \begin{array}{cc} \mathbf{S} & \mathbf{0} \end{array} \right] % \left[ \begin{array}{cc} \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} & \color{red} {\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}} \end{array} \right]^{*} % $$ The product matrices are $$ \begin{align} % \mathbf{A}^{*}\mathbf{A} &= % \left[ \begin{array}{cc} \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} & \color{red} {\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}} \end{array} \right] % \left[ \begin{array}{cc} \mathbf{S}^{2} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right] % \left[ \begin{array}{cc} \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} & \color{red} {\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}} \end{array} \right]^{*} \\ % \mathbf{A}\mathbf{A}^{*} &= % \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} \, \, \mathbf{S}^{2} \, \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} \\ % \end{align} $$

For the hermitian matrix, $$ \begin{align} \mathbf{A} &= \mathbf{A}^{*} \\ \mathbf{U} \, \Sigma \, \mathbf{V}^{*} &= \mathbf{V} \, \Sigma \, \mathbf{U}^{*} \end{align} $$ because in this case $\Sigma = \Sigma^{\mathrm{T}}$.

Answer1: I think your statement above the two questions is correct. But only when A is square, symmetric, positive definite, we can use Cholesky decomposition so that A= BB^T, then B can be decomposed using SVD: B= U \Sigma U^T, thus A=U\Sigma^2U^⊤