Show trace is zero

Solution 1:

It appears from the context in the book that the correct problem is $$ A^2 + B^2 = A B + B A = 0. $$ The middle step is that $(B-A)^2 = 0,$ so we name the nilpotent matrix $N=B-A.$ Wait, I think that is enough. Because it is also true that $(A+B)^2 = 0.$ So $A+B$ and $B-A$ both have trace $0.$ So $tr \; \; 2B = 0.$ That finishes characteristic other than 2. We don't need full Jordan form for nilpotent matrices, just a quick proof that $N^2 = 0$ implies that the trace of $N$ is zero. Hmmm. This certainly does follow from the fact that a nilpotent matrix over any field has a Jordan form, but I cannot say that I have seen a proof of that.

Alright, in characteristic 2 this does not work, in any dimension take $$ A = B, $$ $$ A = B \; \; \; \mbox{then} \; \; A^2 + B^2 = 2 A^2 = 0, \; AB + BA = 2 A^2 = 0. $$

In comparison, the alternate problem $$ A^2 + B^2 = A B + B A = I $$ has the same thing about nilpotence, however in fields where $2 \neq 0$ and $2$ is a square we get a counterexample with $$ A \; = \; \left( \begin{array}{rr} \frac{1}{\sqrt 2} & \frac{-1}{2} \\ 0 & \frac{1}{\sqrt 2} \end{array} \right) $$ and $$ B \; = \; \left( \begin{array}{rr} \frac{1}{\sqrt 2} & \frac{1}{2} \\ 0 & \frac{1}{\sqrt 2} \end{array} \right) $$

Solution 2:

I will prove this just for the case $2\times 2$ being the approach similar for $n\times n$. Any matrices can be written through the generators of U(2) group as $$ A=a_0I+a_1\sigma_1+a_2\sigma_2+a_3\sigma_3=a_0I+{\bf a}\cdot{\bf \sigma} $$

$$ B=b_0I+b_1\sigma_1+b_2\sigma_2+b_3\sigma_3=b_0I+{\bf b}\cdot{\bf \sigma} $$ having $tr(\sigma_i)=0$, $\sigma_i\sigma_j+\sigma_j\sigma_i=2\delta_{ij}I$, $\sigma_i\sigma_j-\sigma_j\sigma_i=2i\epsilon_{ijk}\sigma_k$ and $\sigma_i^2=I$. Besides, we assume $det A, det B\ne 0$. Now we have $$ AB+BA=2a_0b_0+2(a_0{\bf b}\cdot{\bf \sigma}+b_0{\bf a}\cdot{\bf \sigma})+2{\bf a}\cdot{\bf b}=0. $$ and so, it must be $a_0=b_0=0$ and ${\bf a}\cdot{\bf b}=0$. Similarly, one has $$ A^2+B^2=(|{\bf a}|^2+|{\bf b}|^2)I $$ and then $|{\bf a}|^2+|{\bf b}|^2=1$. This implies $tr(A)=tr(B)=0$. These matrices are orthogonal in some sense.