What changes in the representation theory of real Lie algebras?

I read (and hopefully understood) the classification of semi-simple complex Lie algebras, construction of their root spaces and their representation theory (represented as complex matrices). As an example, I am interested in the difference between $\mathfrak{sl}(2)$ and $\mathfrak{su}(2)$.

The standard approach for a complex semi-simple Lie algebra $\mathfrak{g}$ is to construct a Cartan subalgebra $\mathfrak{h}\subset\mathfrak{g}$ and then compute root spaces spanned by eigenvectors of the Cartan subalgebra in the adjoint representation. So far, so good.

Now I want to consider a real semi-simple Lie algebra $\mathfrak{g}$. My understanding was that I can still construct a Cartan subalgebra $\mathfrak{h}\subset\mathfrak{g}$ spanned by a maximal commuting set. By construction (?), the adjoint representation $\mathrm{ad}_H(A)=[H,A]$ for $H\in\mathfrak{h}$ will be anti-symmetric with respect to the Killing form (maybe I'm wrong, because in the real basis the Killing form may not be definite?) which leads to purely imaginary (or vanishing) eigenvalues and generally complex eigenvectors. This means the root spaces cannot be spanned by elements of the real Lie algebra, we need to complexify to get $E_{\pm\alpha}$.

When I now consider a complex representation $\rho$ of the real Lie algebra $\mathfrak{g}$, I would just proceed as for the complex case. I have my complex $E_{\pm\alpha}$ from which I can construct real objects according to $Q_{\alpha}=E_{\alpha}+E_{-\alpha}$ and $P_{\alpha}=i(E_{\alpha}-E_{-\alpha})$. At least for finite dimensional (or discrete) representations, I can use the weight eigenspaces as orthonormal basis of the complex representation vector space $\mathcal{H}$. When I then represent my real Lie algebra element, i.e., Cartan elements $\rho(H)$ or general $\rho(Q_{\alpha})$ and $\rho(P_{\alpha})$, I can decompose them into the $\rho(E_{\pm\alpha})$ by analytic continuation, where I know exactly how $\rho(E_{\pm\alpha})$ raises/lowers the weight...


If I understand correctly, you are interested in representations of a semisimple real Lie algebra $\mathfrak{g}$ on complex vector spaces. These are in equivalence with the complex representations of its complexification $\mathfrak{g}_{\mathbb C}$, cf. Obtaining representation of a real Lie algebra from the complexification "by restriction" , In what sense are complex representations of a real Lie algebra and complex representations of the complexified Lie algebra equivalent?, How does one define weights for a semisimple Lie group?.

In particular, e.g. the representations of both $\mathfrak{su}_2$ and $\mathfrak{sl}_2(\mathbb R)$ are direct sums of irreducible ones, each of which is described up to isomorphism by its highest weight, which are parametrized by $\mathbb Z_{\ge 0}$. They are indeed the irreps of $\mathfrak{sl}_2(\mathbb C)$, which I assume you know well, restricted to the respective real subalgebras.

Note however that this equivalence of categories misses some subtler points, cf. https://math.stackexchange.com/a/3258221/96384. Also, the subsection "a slightly different example" there goes through explicit representations of all three real forms of the complex Lie algebra $\mathfrak{sl}_3(\mathbb C)$ and might be helpful to get a feeling for what's going on.

For the theory so far, it does not matter whether you choose a Cartan subalgebra in the real Lie algebra or in its complexified version, because for the construction of the roots and weights you (should) only use the latter anyway; note that in a real Lie algebra, a CSA plays the role of a maximal torus, but in a complex Lie algebra, that of a maximal split torus, cf. Are there common inequivalent definitions of Cartan subalgebra of a real Lie algebra?; also note that e.g. in $\mathfrak{su}_2$, every $1$-dimensional subspace is a CSA, but there are no roots -- the root spaces only appear in the complexified version.

Finally, in your last paragraph you seem to be mixing up things or start doing something else, namely construct representations on real vector spaces. That is something which needs much more subtle considerations. For the compact real forms, there is a combinatorial criterion about which of the complex representations have "a real structure" (i.e. come from a representation on a real vector space) and which don't, cf. https://math.stackexchange.com/a/2774741/96384 -- e.g. for $\mathfrak{su}_2$, the irreps with even dimension are "truly complex", whereas the ones with odd dimension restrict to acting on real vector spaces. For the non-split and non-compact forms, the story is more intricate. I tried to outline the way I understand it here: https://math.stackexchange.com/a/3298058/96384 (where I think that the accepted answer to the question contains a mistake), and applied that to an example here: https://math.stackexchange.com/a/3298176/96384.


Added: As to what you try in your last paragraph, that would seem to work for the Lie algebras of compact forms, but not in general. A crucial part in the classification is how complex conjugation acts on the roots (and then more refined, on the root spaces). In the compact case, it sends each $\alpha$ to $-\alpha$, and maybe you can choose the $E_\alpha$ so that it sends $E_\alpha$ to $E_{-\alpha}$, meaning your proposed $P_\alpha$ and $Q_\alpha$ are indeed elements of your real Lie algebra. However, the action of complex conjugation on non-compact forms can be very different. (Well for the split forms it's trivial, but ...) To see what can happen, look at the quasi-split form of $\mathfrak{sl}_5(\mathbb C)$. This consists of those matrices in $\mathfrak{sl}_5(\mathbb C)$ where $a_{ij}=-\overline{a_{6-j,6-i}}$ ("antihermitian to the secondary diagonal"). If you call $E_{\alpha_i} := E_{i,i+1}$ for $i=1,…,4$, then observe that complex conjugation transposes $E_{\alpha_1} \leftrightarrow E_{\alpha_4}$ and $E_{\alpha_2} \leftrightarrow E_{\alpha_3}$. (The conjugation operates as the outer automorphism on the Dynkin diagram of $A_4$). So now of course you can still look at "refined" operators playing the role of your $P$'s and $Q$'s (denoting complex conjugation by $\sigma$)

$$E_{\alpha} + \sigma(E_\alpha)$$

$$ iE_{\alpha} + \sigma(i E_{\alpha})= iE_{\alpha} – i \sigma(E_{\alpha})$$

and maybe you can get something like $\sigma(E_\alpha) = \pm E_{\sigma(\alpha)}$ but what exactly happens further depends on what exact $\alpha$ you’re looking at; here, $\alpha_3$ needs different treatment than the other $\alpha$'s. Also, all those refined $P$'s and $Q$'s together might not yet give a basis for the real Lie algebra! Further, what do we do with the $H_\alpha$? Not saying this is not doable, just that it might be quite intricate.