Finding the orbits of $SL(n+1)$ and a parabolic subgroup $P$ in a representation?

Even for reductive Lie groups $G$, the general problem of determining the $G$-orbit decomposition of a linear representation of $G$ is very difficult---or even “hopeless”, see this MathOverflow post. (Some general results are available, however, for nice classes of representations, e.g., where the action is multiplicity-free, as it is in our case, or the representation is a prehomogeneous vector space for $G$.)

In our situation the problem is tractable but already a little complicated. So, I'll describe the analogous complex case and then mention the refinements needed for the real case that the question asked about: We take $G := SL(n + 1, \Bbb C)$ and $P$ the stabilizer of a ray in the standard representation $\Bbb C^{n + 1}$ of $SL(n + 1, \Bbb C)$. (Normal, regular) parabolic geometries of the corresponding type $(G, P)$, $n > 1$, correspond to complex projective structures of (complex) dimension $n$.

First recall the following facts: For any quadratic form $q$ on $\Bbb C^{n + 1}$ there is a basis in which $q$ has the form $$q(X) = (X^1)^2 + \cdots + (X^r)^2 .$$ The invariant $r$ is independent of the basis chosen, so if for two quadratic forms $q, q'$ the respective invariants $r, r'$ satisfy $r \neq r'$, then $q$ and $q'$ are in different orbits. On the other hand, some linear algebra shows that $r = r'$ implies that $q$ and $q'$ are in the same orbit in the usual action of $GL(n, \Bbb C)$ on quadratic forms, that is, that $r$ is essentially the only invariant of the action. So, the orbits of the $GL(n + 1, \Bbb C)$ are the sets $\mathcal O_r$, respectively consisting of the symmetric bilinear forms whose invariant has value $r$.

Now, we can identify quadratic forms on $\Bbb C^{n + 1}$ with symmetric bilinear forms via polarization, in which case the invariant $r$ for a bilinear form $h \in S^2 \Bbb C^{n + 1}$ is just the rank of the map $\Bbb C^{n + 1} \mapsto (\Bbb C^{n + 1})^*$, $X \mapsto h(X, \,\cdot\,))$, or equivalently, the rank of the matrix representation $[h]$ of $h$ with respect to any basis. The orbit $\mathcal O_{n + 1}$ consists precisely of the nondegenerate forms.

Now, we consider the restriction of this $GL(n + 1, \Bbb C)$-action to $SL(n + 1, \Bbb C)$, which is precisely the subgroup that fixes a volume form $\omega \in \Lambda^{n + 1} \Bbb C^*$ under the induced action on that vector space. In particular, this means that $\mathcal O_{n + 1}$ splits into $SL(n + 1, \Bbb C)$-orbits: A nondegenerate bilinear form $h$ determines a volume form $\Omega$ up to sign. This pair is characterized by $\Omega(E_1, \ldots, E_{n + 1}) = \pm 1$ for any basis $(E_a)$ orthonormal with respect to $h$, and $\Omega = \pm \lambda \omega$ for some $\lambda \neq 0$ determined up to sign, and by construction the pair is invariant under the $SL(n + 1, \Bbb C)$-action.

The remaining orbits do not split under our restriction, however. Given two symmetric bilinear forms with invariant $r < n + 1$, we can choose bases in which the induced quadratic forms have the canonical form in the above display equation, in which case they are related by some element $$A = \pmatrix{B&C\\0&D}, $$ where the submatrix $A'$ has size $r \times r$. By construction, those two forms are also related by the matrix $A = \pmatrix{B&C\\0&D'}$ for any $C' \in GL(n + 1 - r, \Bbb C)$. In particular, we can choose such a matrix so that $\det A = 1$.

So, if we fix a reference volume form $\omega$ the orbits of the $SL(n + 1, \Bbb C)$-action on $S^2 (\Bbb C^{n + 1})^*$ are

• the sets $\mathcal O_{n + 1, \Lambda}$ of nondegenerate bilinear forms with volume forms $\pm \lambda \Omega$, for each (nonzero) pair $\Lambda = \{\pm \lambda\} \subset \Bbb C$, and
• the sets $\mathcal O_r$, $r < n + 1$.

We now find the $P$-orbit decomposition, which must refine the $SL(n + 1, \Bbb R)$-orbit decomposition. Since $P$ is characterized by stabilizing a preferred ray $\mathbf R$ in $\Bbb C^{n + 1}$, these refinements are governed by the interaction of the bilinear forms with that ray.

Given a nondegenerate bilinear form $h \in \mathcal O_{n + 1, \Lambda}$, the action of $SO(h)$ on the space of rays in $\Bbb C^{n + 1}$ has two orbits: The set of isotropic rays and the set of nonisotropic rays. If $\mathbf R$ is isotropic, we can choose a basis $(E_a)$ such that $E_1 \in \mathbf R$, $E_2, \ldots, E_n \in {[\mathbf R]}^{\perp}$, $E_{n + 1} \not\in [{\mathbf R}]^{\perp}$ and $(E_2, \ldots, E_n)$ descends to an oriented orthonormal basis of $[{\mathbf R}]^{\perp} / [{\mathbf R}]$. Now, $P \cap SO(h) < P$ (which is actually a parabolic subgroup of $SO(h)$) acts transitively on the space of such bases, which implies that the nondegenerate bilinear forms with $\lambda = 1$ with respect to which $\mathbf R$ is isotropic comprise a $P$-orbit. If $\mathbf R$ is nonisotropic, we can pick a basis $(E_a)$ orthogonal with respect to $h$ such that $E_1 \in {\mathbf R}$, and the group $P \cap SO(h) \cong SO(n, \Bbb C) < P$ acts transitively on the set of such bases, so the nondegenerate bilinear forms with given $\Lambda$ with respect to which $\mathbf R$ is nonisotropic comprise a $P$-orbit.

Finally, consider a degenerate bilinear form $h \in \mathcal O_r$. We can proceed roughly as in the nondegenerate case, but this time (for $r > 0$) there are three cases, according to whether (1) $\mathbf R$ is contained in the null space $N$ of $h$, (2) $\mathbf R$ is not contained in the image of the above map $\Bbb C^{n + 1} \to (\Bbb C^{n + 1})^*$ but is still isotropic with respect to $h$ (that is, whether the restriction of $h$ to $\mathbf R$ is zero), and (3) $\mathbf R$ is not isotropic with respect to $h$. I haven’t worked out the details, but I expect that we can proceed as for the nondegenerate case to show that each $r > 0$ each of these three cases corresponds to a single orbit; of course, the orbit $\mathcal O_0$ consists just of the zero bilinear form.

The real case is similar, except that because of the presence of signature, the existence of the above canonical form for $q$ is replaced by the statement of Sylvester's Law of Inertia: For any symmetric bilinear form on $\Bbb R^{n + 1}$ there is a basis in which $$h(X, Y) = \sum_{i = 1}^{n + 1} \varepsilon_i X_i Y_i$$ for coefficients $\varepsilon_i \in \{-1, 0, +1\}$. Thus, the $GL(n + 1, \Bbb R)$-orbit decomposition is correspondingly finer, and we get a single orbit for each ordered partition of $n + 1$ into three nonnegative integers. In the analysis of the orbit decomposition for (the real) P, the cases in which the preferred ray is nonisotropic split further according to whether it is spacelike or timeline.