Is it necessary that $\det(AB-BA)= 0$?
Suppose $A^2 + B^2 = 2AB$ where $A,B$ are square matrices, what can be said about $\det(AB-BA)$?
My method : subtracting $-BA $ to both sides we get $[\det(A-B)]^2 = \det(AB-BA)$, now is it possible that this can be simplified to get $\det$ is singular/nonsingular?
This question was asked (for the first time ? Probably not) as problem 3 of this romanian olympiad. Here is a translation/adaptation of the solution from Romanian into English (by a Frenchman...):
Relationship $A^2+B^2=2AB$ can be given the two following equivalent expressions:
$$(A − B)^2 = AB − BA\tag{1}$$ $$A(A − B) = (A − B)B\tag{2}$$
Let us assume that $AB−BA$ is invertible.
As a consequence, using (1), $A−B$ would itself be invertible (think to non-zero determinant). Now using (2), we get
$$B = (A − B)^{−1}A(A − B)$$
which gives:
$$A-B = A-(A − B)^{−1}A(A − B)\tag{3}$$
Right-multiplying (3) by $ (A − B)^{−1}$ gives
$$I_n = A(A − B)^{−1} − (A − B)^{−1}A$$
Now taking the trace of both sides, we obtain a contradiction ¸i.e.,:
$$\underbrace{tr(I_n)}_n = \underbrace{tr(A(A − B)^{−1} − (A − B)^{−1}A)}_0$$
(due to the classical result $tr(AB)=tr(BA)$).
From this contradiction, we deduce that our assumption of inversibility of $AB − BA$ cannot hold.
Remark: There was a second question asking to show that if $rank(A-B)=1$, then matrices $A$ and $B$ commute.
Ok, this may be an overkill. Assume the underlying field has characteristic zero. The given condition implies that $(A-B)^2=[A,B]=[A-B,B]$, where the symbol $[X,Y]$ denotes the commutator $XY-YX$. It follows that $A-B$ commutes with $[A-B,B]$. Therefore $[A-B,B]$ is nilpotent, by Jacobson's lemma. Yet, $[A-B,B]$ is identical to $[A,B]$. Hence $[A,B]$ is nilpotent.