$\operatorname{tr}(AABABB) = \operatorname{tr}(AABBAB)$ for $2×2$ matrices

First if $N$ is $2\times2$ matrix with $\operatorname{tr}(N)=0$, then $N^2$ is scalar (either $N$ is nilpotent or its eigenvalues are opposite and have same square). This implies that $\operatorname{tr}(N^3)=0$.

And $$\begin{eqnarray} (AB-BA)^3 &=& (ABABAB - BABABA) + (ABBABA + BABAAB + BAABBA) \\ && - (ABABBA + ABBAAB + BAABAB) \end{eqnarray} $$

taking the trace this gives

$$0 = 0 + 3\operatorname{tr}(ABBABA) - 3 \operatorname{tr}(ABABBA) = 3 \operatorname{tr}(AABBAB - AABABB).$$


Also relevant is the Amitsur-Levitki theorem.


I recall reading long ago that such trace identities generally arise from those associated with the Cayley-Hamilton theorem (by multilinearizing the characteristic polynomial[2]). A quick web search on related keywords turned up the following paper[1]. In the introduction it is stated that "we prove that all trace identies of the full matrix algebra of order n over a field of characteristic zero are consequences of one corresponding to the Hamilton-Cayley theorem". There is also independent seminal work of Procesi, who obtains the trace identities via multilinear invariants of tensor products of vector spaces. No doubt much work has been done the in three decades since this seminal work appeared. A search on Razmyslov / Procesi and "trace identities" reveals much.

[1] Razmyslov. Trace identities of full matrix algebras over a field of characteristic zero.
Math Ussr Izv, 1974, 8 (4), 727-760.

[2] Formanek. Polynomial identities and the Cayley-Hamilton Theorem.
The Mathematical Intelligencer. Vol. 11, 1, 1989, 37-39.