When is $A^TBA$ invertible, where $B$ is an invertible, symmetric matrix?

Let $B$ be an $n×n$ invertible matrix such that $B^T=B$, and let $A$ be a $n×2$ matrix with linearly independent columns. When is the product $A^TBA$ invertible?

Solution 1:

We'll prove $A^TBA$ is invertible $\iff$ $B\left(\text{Col}(A)\right)\cap \text{Col}(A)^{\perp}=\{0\}$.

Here is the "only if" part. Assume $A^TBA$ is invertible but there is some non$-$zero $y$ in $B\left(\text{Col}(A)\right)\cap \text{Col}(A)^{\perp}$. Then $y=BAx$ for some $x\in \mathbb{R}^2$. Because $\text{Col}(A)^{\perp}=\text{Nul}(A^T)$ we also get $A^Ty=0$. Putting both together implies $A^TBAx=0$ i.e. $x\in \text{Nul}(A^TBA)$. Now $A^TBA$ is assumed to be invertible, so $x$ must be the zero vector, and so is $y$, a contradiction.

Here is the "if" part. Assume $B\left(\text{Col}(A)\right)\cap \text{Col}(A)^{\perp}$ has a trivial intersection and $A^TBAx=0$ for some $x\in \mathbb{R}^2$. Then we have $BAx\in \text{Nul}(A^T)=\text{Col}(A)^{\perp}$. But $BAx$ also belongs to $B\left(\text{Col}(A)\right)$ which means $BAx=0$ from our hypothesis. Hence $Ax=B^{-1}0=0$ and since $A$ has independent columns, $x=0$ and $A^TBA$ is invertible.