determinant inequality $\det(A^2+AB+B^2)\geq\det(AB-BA)$

Solution 1:

$\square$ Since for any $2\times 2$ matrix $M$ one has $$2\operatorname{det}M=\left(\operatorname{Tr}M\right)^2-\operatorname{Tr}M^2,\tag{1}$$ the inequality we want to prove is equivalent to \begin{align}\left(\operatorname{Tr}(A^2+B^2+AB)\right)^2&\geq \operatorname{Tr}\left(\left(A^2+B^2+AB\right)^2-\left(AB-BA\right)^2\right)=\\ &=\operatorname{Tr}\left(A^4+B^4-ABAB+2A^3B+2AB^3+4A^2B^2\right). \tag{2}\end{align} On the other hand, using (1) to rewrite the inequality proved here (mentioned in the comments) and replacing therein $B\rightarrow -B$, we obtain exactly the same inequality (2). $\blacksquare$

Solution 2:

As invertible matrices are dense in the matrix space, we may assume that $A$ is invertible. Left- and right- multiply both sides by $\det(A^{-1})$, the inequality in question becomes $$\det(I + BA^{-1} + A^{-1}B\,BA^{-1}) \ge \det(BA^{-1} - A^{-1}B).\tag{1}$$ So, it suffices to prove that $$\det(I + X + YX) \ge \det(X-Y)\tag{2}$$ for any invertible $X,Y$ that are similar to each other. Let $X=kI+X_0$ and $Y=kI+Y_0$, where $X_0$ and $Y_0$ are two nilpotent matrices. Then $(2)$ can be further rewritten as $$\det\left((1+k+k^2)I + (k+1)X_0 + kY_0 + Y_0X_0\right) \ge \det(X_0-Y_0).\tag{3}$$ If $X_0=Y_0=0$, the inequality is trivial. Suppose $X_0, Y_0$ are nonzero. So, by a change of basis, we may assume that $$ Y_0=\pmatrix{0&1\\ 0&0},\ X_0=\pmatrix{p&-q\\ r&-p}$$ with $qr=p^2$. If one calculates both sides of $(3)$ directly using the entries of $X_0$ and $Y_0$, one will find that the LHS is equal to $(1+k+k^2)^2 + r$ and the RHS is $r$. Now the inequality follows immediately.

Solution 3:

For any two $2\times2$ matrices $A$ and $B$, the following identity holds: $\renewcommand{\tr}{\operatorname{tr}}$ $\renewcommand{\adj}{\operatorname{adj}}$ $$ \det(X+Y) \equiv \det(X) + \det(Y) + \tr(X\adj(Y)).\tag{$\ast$} $$ Therefore, \begin{align} \det(AB-BA) &=2\det(AB) + \tr(AB\adj(-BA))\\ &=2\det(AB) - \tr(AB\adj(A)\adj(B)). \end{align} Write $t=\tr(A\adj(B))=\tr(B\adj(A))$. Then \begin{align} &\det(A^2+AB+B^2)\\ =&\det((A+B)^2 - BA)\\ =&\det((A+B)^2) + \det(-BA) + \tr((A+B)^2\adj(-BA))\quad\text{ by } (\ast)\\ =&\det(A+B)^2 + \det(AB) - \tr((A^2+B^2+AB+BA) \adj(A)\adj(B))\\ =&\det(A+B)^2 + \det(AB) - (\det(A)+\det(B))t - \tr(AB \adj(A)\adj(B)) - 2\det(AB)\\ =&\left(\det(A)+\det(B)+t\right)^2 - 3\det(AB) - (\det(A)+\det(B))t + \det(AB-BA)\\ =&\left(t + \frac{\det(A)+\det(B)}2\right)^2 + \frac34\left(\det(A)-\det(B)\right)^2 + \det(AB-BA)\\ \ge&\det(AB-BA). \end{align}