Inscrutable proof in Humphrey's book on Lie algebras and representations
Solution 1:
This lemma troubled me until I saw proof of Cartan's criterion for complex case in an appendix of Fulton & Harris book.
Denote $x_s = d(\lambda_1, \ldots, \lambda_n)$ the semisimple part of $x$, and $\bar{x}_s = d(\bar{\lambda}_1, \ldots, \bar{\lambda}_n)$, the matrix you get when you ''complex-conjugate'' $x_s$. Lemma is then replaced with the following simple observation: $\operatorname{Tr}(x \bar{x}_s) = |\lambda_1|^2 + \ldots + |\lambda_n|^2 = 0$ implies $x$ nilpotent.
For completeness, proof of Cartan's criterion using this fact:
Let $V$ be a finite-dimensional complex vector space, $L$ Lie subalgebra of $\mathfrak{gl} V$.
Let's assume that $\operatorname{Tr}(xy) = 0$, for all $x \in [LL], y \in L$. If we show that every $x \in [LL]$ is nilpotent, it will follow (by Engel's theorem) that $[LL]$ is nilpotent and that will imply that $L$ is solvable. We'll do this using the aforementioned simple observation. Write $x = \sum_i [y_i, z_i]$. $$\operatorname{Tr}(x\bar{x}_s) = \sum_i \operatorname{Tr}([y_i, z_i]\bar{x}_s) = \sum_i \operatorname{Tr}(y_i, [z_i, \bar{x}_s])$$ Now, we can use Lagrange's interpolation to write $\operatorname{ad}(\bar{x}_s)$ as a polynomial in $\operatorname{ad}(x_s)$ without constant term, so it follows that $[z_i, \bar{x}_s]$ is an element of $[L, L]$. Now, our assumption gives us $\operatorname{Tr}(x \bar{x}_s) = 0$.
As you can see, more general proof follows this one closely. Using the nice properties of complex conjugation, we don't have to check $\operatorname{Tr}(xy) = 0$ for all $y \in M$ to get $x$ nilpotent, we just have to check that for one element in $M$, namely $\bar{x}_s$. I can imagine Cartan proving first the complex case, then generalizing the proof to the one you read in Humphreys.