$A$ and $B$ are $3\times 3$ real matrices such that $\operatorname{rank}(AB)=1$, then $\operatorname{rank}(BA$) can not be which of the following?

I was thinking about the problem that says:

If $A$ and $B$ are $3\times 3$ real matrices such that $\operatorname{rank}(AB)=1$, then $\operatorname{rank}(BA)$ can not be which of the following?
(a) $0$
(b) $1$
(c) $2$
(d) $3$.

My attempt: I have chosen suitable $3 \times 3 $ matrices for $A$ and $B$ keeping in mind that $\operatorname{rank}(AB)=1$. Say for example if I take $A$ and $B$ to be $$A = \begin{pmatrix} 1 &2 &0 \\ 0 & 0 &0 \\ 0 & 0 &0 \end{pmatrix}$$ and $$B = \begin{pmatrix} -2 &1 &0 \\ 1 & 0 &0 \\ 0 & 0 &0 \end{pmatrix}$$
respectively, then I see $\operatorname{rank}(AB) = \operatorname{rank}(BA) = 1$. So, option (b) can not be correct. Do I have to keep choosing the matrices and then observe which of the option holds good. Is this kind of approach right to tackle the problem? I am looking for a direct way (e.g. application to some theorem) which can give me the result. I have also noticed that $AB$ and $BA$ are similar matrices as we see that $A^{-1}(AB)A=BA$. Is this observation going to help me in any way? Thanks in advance for your time.

Solution 1:

If you only want to solve the problem, then as the others have pointed out, (d) is impossible and hence the answer is (d). It is helpful, however, to constuct some examples to convince yourself that (a)-(c) are really possible (this is also a sanity check against misprints). For this purpose, you may consider the matrices $$ A=\begin{pmatrix}1\\&1\\&&0\end{pmatrix},\ B=\begin{pmatrix}b&0&1\\0&0&0\\0&c&0\end{pmatrix}. $$ Then $\operatorname{rank}(AB)=1$. Now you may try to choose some appropriate values of $b$ and $c$ to make $\operatorname{rank}(BA)$ equal to $0,1$ or $2$.

Solution 2:

According to these properties of ranks listed on Wikiepdia : http://en.wikipedia.org/wiki/Rank_(linear_algebra)

If $A$ is $m \times n$

1. $\operatorname{rank}(AB) \leq \min(\operatorname{rank}(A), \operatorname{rank}(B))$ if $B$ is $n \times k$
2. $\operatorname{rank}(AB) = \operatorname{rank}(A)$ if $B$ is $n \times k$ of rank $n$

Note that in your situation $m$, $n$ and $k$ is $3$

By property (1) the only way to have $\operatorname{rank}(BA) = 3$ is to have both $\operatorname{rank}(A) = 3$ and $\operatorname{rank}(B) = 3$. Since $\min(\operatorname{rank}(A), \operatorname{rank}(B)) = \min(\operatorname{rank}(B), \operatorname{rank}(A))$.

But in that situation, we can apply property (2): $B$ is $3 \times 3$ of rank $3$ and so $\operatorname{rank}(AB) = \operatorname{rank}(A) = 3$.

So you can't have at the same time $\operatorname{rank}(BA) = 3$ and $\operatorname{rank}(AB) \neq 3$ using $3 \times 3$ matrix only.

Thus answer (d) is correct.

Solution 3:

The answer is (d), i.e. if $rank(AB)=1$, then $rank(BA)$ cannot be $3$. Let's prove it by contradiction. Suppose $A, B$ are $3\times 3$ matrices such that $rank(AB)=3$, then $AB$ is invertible. This implies that $A$ and $B$ are invertible (otherwise, if $A$ is not invertible, then $A$ could not be full rank, i.e. $rank(A)<3$. This would imply that $rank(AB)\leq rank(A)<3$, which contradicts $rank(AB)=3$. Similarly you can prove that $B$ is invertible). Then $BA$ is invertible since product of invertible matrices is invertible. This implies that $rank(BA)=3$, which contradicts to the assumption that $rank(AB)=1$.

To see that (a), (b), (c) are possible, we can construct some examples of $A$ and $B$. As you have shown that (b) is possible (In fact, we can just take $A$ to be any rank 1 matrix, i.e. $rank(A)=1$ and $B$ to be the identity. Then $AB=BA=A$ has rank $1$). I will leave other cases to you.