Given $BA$, find $AB$.

Solution 1:

For any square matrix $S$, let us denote its minimal polynomial by $m_S$ and denote its characteristic polynomial by $p_S$.

Facts:

1. A square matrix is diagonalizable (over $\Bbb C$) iff its minimal polynomial has no multiple root.
2. If $S$ is a diagonalizable square matrix of order $n$ and if $\lambda$ is an eigenvalue of $S$ with multiplicity $k$, then the rank of $\lambda I_n-S$ is $n-k$.
3. Let $A$ and $B$ be matrices of size $m\times n$ and $n\times m$ respectively. If $q(BA)=0$ for some polynomial $q$, then $AB\cdot q(AB)=A\cdot q(BA)\cdot B=0$.
4. Moreover, if $m\ge n$ additionally, then $p_{AB}(\lambda)=\lambda^{m-n}p_{BA}(\lambda)$.

In your question, as implicitly shown in user1551's answer, $$m_{BA}(\lambda)=(\lambda-1)(\lambda-2),\quad p_{BA}(\lambda)=(\lambda-1)^2(\lambda-2).$$ Then from 3. and 4. (and the fact that eigenvalues are roots of minimal polynomial) we know $$m_{AB}(\lambda)=\lambda(\lambda-1)(\lambda-2),\quad p_{AB}(\lambda)=\lambda(\lambda-1)^2(\lambda-2).$$ Then from 1. and 2. we know $AB$ is diagonalizable and the rank of $I_4-AB$ is $2$.

As a result, the determinant of any $3\times 3$ submatrix of $I_4-AB$ is $0$, which implies that $x=0$.

Solution 2:

Hint. By applying appropriate row and column operations on $AB$, we find that if $$ P=\pmatrix{2&0&2&4\\ -1-2x&1&-3x&-1-4x\\ 2&0&3&4\\ 6&0&6&14}, \ P^{-1}=\pmatrix{\tfrac92&0&-1&-1\\ 3&1&x-1&\tfrac{-1}2\\ -1&0&1&0\\ \tfrac{-3}2&0&0&\tfrac12}, $$ then $$ (PA)(BP^{-1}) = \pmatrix{0&0&0&0\\ 0&1&x&0\\ 0&0&1&0\\ 0&0&0&2}.\tag{1} $$ Let $PA=\pmatrix{a^T\\ A_1}$ and $BP^{-1}=\pmatrix{b&B_1}$ where $a,b\in\mathbb{R}^3$ and $A_1,B_1\in M_3(\mathbb{R})$. Since $BA$ is invertible, we must have $\operatorname{rank}(PA)=\operatorname{rank}(BP^{-1})=3$ (why?). Hence argue that $a=b=0$ and $A_1B_1=\pmatrix{1&x&0\\ 0&1&0\\ 0&0&2}$. Yet, $A_1B_1$ is similar to $B_1A_1$, and $B_1A_1=BA$ is diagonalisable (exercise). Hence there is only one possible value of $x$, which is ...