$\operatorname{SO}(n)$ is an (abstractly) maximal subgroup of $\operatorname{SL}(n)$

Solution 1:

$n=2$. A geometric proof: $\mathrm{SL}_2(\mathbf{R})$ acts on the hyperbolic plane as the full orientation-preserving isometry group with stabilizer $\mathrm{SO}(2)$. We need to check this action is primitive. Indeed, let $R$ be an invariant equivalence relation, not reduced to equality. Since the action is transitive on pairs at given distance, there exists a subset $M$ of $\mathbf{R}_{\ge 0}$, not reduced to $\{0\}$, such that $R$ equals the set of pairs at distance in $M$. Choose $r>0$ in $M$. Since any two points $x,y$ in the hyperbolic plane can be joined by a discrete path $x=x_0,\dots,x_n=y$ with $d(x_i,x_{i+1})=r$, we deduce that $xRy$. Hence $R$ is the indiscrete equivalence relation.

Now consider $n\ge 3$. Let $H$ be a subgroup properly containing $\mathrm{SO}(n)$. Let $D$ be the set of diagonal determinant 1 matrices with positive entries. Let $D_H$ be the intersection $D\cap H$. It is enough to show that $D_H=D$ (indeed, by polar decomposition it is enough to check that every positive determinant 1 symmetric matrix belongs to $H$, and using ortho-diagonalization, it is enough to check the latter for diagonal matrices). Moreover, again by the same polar decomposition argument, the assumption $H\neq\mathrm{SO}(n)$ implies that $D_H$ is not reduced to $\{1\}$.

Note that $D_H$ is stable under permutation of coordinates. Hence it contains some diagonal matrix $(t_1,t_2,t_3,\dots)$ with $0<t_2<1<t_1$. Hence it also contains the diagonal matrix $(t_2,t_1,t_3,\dots,t_n)$. Hence it contains the diagonal matrix $(u,u^{-1},1,\dots,1)$ with $u=t_1/t_2>1$. By the case $n=2$ applied to the $2\times 2$ north-west block, it therefore contains all matrices $(v,v^{-1},1,\dots,1)$. Hence, it also contains all its conjugates by permutations of coordinates. Since any element $(u_1,\dots,u_n)$ of $D$ (so $\prod u_i=1$) is the product of the diagonal matrices $(1,\dots,v_i,v_i^{-1},1,\dots,1)$ for $i=1\dots n-1$ with $v_i=u_1\dots u_i$, we deduce $D_H=D$, which finishes the proof.

Finally the most subtle case is $n=2$. In this case one can try a more direct approch. Namely write $K=\mathrm{SO}(2)$. By assumption there is a non-trivial element $g$ in $D_H$. I'd try to show that the map $K^3\to\mathrm{GL}_2(\mathbf{R})$, mapping $(x,y,z)$ to $xgyg^{-1}z$, has its differential of rank 3 somewhere. Indeed, if so, it implies that its image has non-empty interior in $\mathrm{SL}_2(\mathbf{R})$, and hence $H$ has non-empty interior in $\mathrm{SL}_2(\mathbf{R})$, so is an open subgroup, so equals $\mathrm{SL}_2(\mathbf{R})$.