Let $A,B\in M_2(\mathbb{C})$ such that $A^2+B^2=3AB$. Prove or disprove that $ \det(AB+BA)=\det(2AB). $

Solution 1:

The result follows immediately if $AB=BA$. So, let us assume that $AB\ne BA$. By Cayley-Hamilton theorem, $A^2$ is a linear combination of $A$ and $I$ and $B^2$ is a linear combination of $B$ and $I$. Therefore the condition $A^2+B^2=3AB$, or equivalently $AB-\frac13(A^2+B^2)=0$ can be rewritten in the form of $$ (A+pI)(B+qI)=rI.\tag{1} $$ By assumption, $AB\ne BA$. Hence the RHS of $(1)$ must be zero. That is, $XY=0$ when $X=A+pI$ and $Y=B+qI$. Hence one of $X$ or $Y$ is singular but nonzero (it cannot be zero because $AB\ne BA$ implies that $A$ and $B$ are not scalar matrices). Suppose $X\ne0$ is singular (the other case is similar). By a change of basis, we may assume that $X$ is already in Jordan form. However, as \begin{aligned} \pmatrix{0&1\\ 0&0}Y=0\quad\Rightarrow\quad Y=\pmatrix{\ast&\ast\\ 0&0},\\ \underbrace{\pmatrix{\lambda&0\\ 0&0}}_{\lambda\ne0}Y=0\quad\Rightarrow\quad Y=\pmatrix{0&0\\ \ast&\ast}, \end{aligned} $X$ and $Y$ are simultaneously triangularised. Hence $A$ and $B$ are simultaneously triangulable and the result follows.

Remark. Using the technique applied in loup blanc's answer to a similar question, we can prove that the statement here is actually true when $A,B$ are $n\times n$. Let $w=\frac{3+\sqrt{5}}{2}$ and $w'=\frac{3-\sqrt{5}}{2}$. Then we may write $A=wX+w'Y$ and $B=X+Y$ for some $X$ and $Y$. The condition $A^2+B^2=3AB$ thus reduces to $$ XY=kYX,\tag{2} $$ where $0<k=\frac{w'}{w}<1$. Therefore at least one of $X$ or $Y$ is singular. If $X$ is singular (the other case is similar), $(2)$ implies that $\ker(X)$ is an invariant subspace of $Y$. Therefore $X$ and $Y$ share an eigenvector in $\ker(X)$. Continue the argument recursively, we see that $X$ and $Y$ are simultaneously triangulable. Hence $A$ and $B$ are simultaneously triangulable too and the result follows.