$A,B$ be Hermitian.Is this true that $tr[(AB)^2]\le tr(A^2B^2)$?

Solution 1:

Yes. Equality holds iff $AB = BA$.

Hint: Note that $AB - BA$ is skew-Hermitian, and that $$ 2\operatorname{trace}[(AB)^2] - 2\operatorname{trace}(A^2B^2) =\\ \operatorname{trace}(ABAB + BABA -ABBA - BAAB)=\\ \operatorname{trace}[(AB - BA)^2] $$

Note: The inequality assumes that both $\operatorname{trace}[(AB)^2]$ and $\operatorname{trace}(A^2B^2)$ are real. This is fine because the product of Hermitian matrices necessarily has a real trace. Note that $$ \operatorname{trace}[(A+aI)(B+bI)] =\\ \operatorname{trace}(AB) + b\operatorname{trace}(A) + a \operatorname{trace}(B) + abn $$ and that $a,b>0$ may be chosen so that $A +aI$ and $B+bI$ are positive definite. The product of (Hermitian) positive definite matrices always has positive eigenvalues.