Low-dimensional Irreducible Representations of $S_n$

Following Jyricki's excellent suggestion, I will show that, for $n \neq 4$, the $n-2$ dimensional representations of $S_{n-1}$ do not extend to representations of $S_n$. Let $V$ be the vector space of $(n-1)$-tuples of real numbers adding up to $0$, with $S_{n-1}$ acting by permutation. I will show that this does not extend to an action of $S_n$.

Suppose for the sake of contradiction that there is an action of $S_n$ on $V$ extending the action of $S_{n-1}$. Since this is an action of a finite group in characteristic zero, it preserves a symmetric bilinear form. There is only one (up to scaling) $S_{n-1}$-invariant symmetric bilinear form on $V$ -- namely, the one induced from the standard inner product on $\mathbb{R}^{n-1}$. So $S_n$ must preserve this inner product.

For $1 \leq i, j < n$, the transposition $(i j)$ acts on $V$ by orthogonal reflection in the vector $e_i - e_j$. Since $(i n)$ and $(i j)$ are conjugate in $S_n$, we know that $(i n)$ acts by orthogonal reflection in some vector; call it $v_i$.

For $1 \leq j,k < n$, with $i \neq j$, $k$, the reflections $(i n)$ and $(j k)$ commute. This means that $e_j -e_k$ and $v_i$ must be orthogonal. So $v_i$ must be of the form $$v_i = a \cdot \left( \sum_{j \neq i} e_j - (n-2) e_i \right)$$ for some scalar $a$.

Now recall that ${\Large (} (ij) (in) {\Large )}^3 = \mathrm{Id}$. So the angle between $v_i$ and $e_i - e_j$ must be $\pi/3$ or $2 \pi /3$. We compute $$|e_i - e_j| = \sqrt{2}$$ $$|v_i| = a \sqrt{(n-2)+(n-2)^2 } = a \sqrt{(n-1)(n-2)}$$ $$\langle e_i - e_j, v_i \rangle = a \left( -(n-2) - 1 \right) = - a (n-1).$$ So the angle between $v_i$ and $e_i -e _j$ is $$\cos^{-1} \left( \frac{-(n-1)}{\sqrt{2(n-1)(n-2)}} \right).$$

When $n=4$, this is $\cos^{-1} (-\sqrt{3}/2) = 2 \pi /3$. And, indeed, when $n=4$, the action does extend! But for $n>4$, $\frac{(n-1)}{\sqrt{2(n-1)(n-2)}} \neq \pm \sqrt{3}/2$, so the action does not extend.

Now, a quick sketch of how to finish the problem from here. We prove by induction the statement

For $n \geq 5$, the only representations of $S_n$ of dimension $<n$ are direct sums of (a) the trivial rep (b) the sign rep (c) the $n-1$ dimensional rep on $n$-tuples of numbers summing to $0$ and (d) the tensor product of (b) and (c).

The base case is left to you; starting by considering the eigenvalues of the $5$-cycle is a good idea.

Let $W$ be a rep of $S_n$ of dimension $<n$. Consider the restriction of $W$ to $S_{n-1}$. If $\dim W < n-2$, then induction shows that it is a direct sum of (a)'s and (b)'s, and I leave it as an exercise to show that $W$ is the same direct sum as a representation of $S_n$.

If $\dim W = n-2$ and $W$ is not made up of (a)'s and (b)'s, then $W$ is either type (c) or type (d). The above result gives a contradiction in the first case, and tensoring with the sign rep and then applying the above result gives a contradiction in the second case.

Finally, we must deal with the possibilities when $\dim V = n-1$. Details are left to you, but I think modifying the above argument will do this case as well.

Theorem. For $n \geq 7$, the only representations of $S_n$ of dimension $\leq n-1$ are direct sums of (a) the trivial rep (b) the sign rep (c) the $n-1$ dimensional rep on $n$-tuples of numbers summing to $0$ and (d) the tensor product of (b) and (c).

I'll use the induction hypothesis that the statement is true for $S_{n-1}$ and $S_{n-2},$ using $S_7$ and $S_8$ as base cases. I haven't proved the base cases but reliable references say they are true. A similar induction was used in "Braid group representations of low degree‎" by Formanek to prove a result for the braid group.

It suffices to consider only unitary representations $\rho:S_n\to U_m,$ by the unitary trick as in the other answer. And we can assume $\rho$ is irreducible and faithful - otherwise it factors through $\det:S_n\to S_n/A_n.$

Case 1. The multiplicity of $-1$ in $\rho((n-1,n))$ is $1$ or $m-1.$

If the rank is $m-1,$ tensor with $\det$ to reduce to the case where the rank is $1.$ This case is then a representation as a reflection group: each $\rho((i,j))$ is reflection in the hyperplane perpendicular to a unit vector $v_{ij}$ which is determined up to phase.

For any subset $X\subset\{1,\dots,n\}$ with $|X|=n-1,$ the restriction to $S_X$ defines a representation $S_{n-1}\simeq S_X\to U_m.$ This representation is faithful and generated by $n-2$ reflections, so it's rep "(c)" of $S_{n-1}$ if $m=n-2,$ or the direct sum of that rep with a $1$-dim trivial rep if $m=n-1.$ This fixes the inner products between each pair of vectors $v_{ij},$ and hence determines $\rho$ up to isometry as being rep "(c)" of $S_n$ (and forces $m=n-1.$)

Case 2. The multiplicity of $-1$ in $\rho((n-1,n))$ is in $2,\dots,m-2.$

Let $V=V_{-1}\oplus V_1$ be the eigenspace decomposition with respect to $\rho((n-1,n)).$ The restriction of $\rho$ to $S_{n-2}$ (the symmetries of $\{1,\dots,n-2\}$) splits as $V_{-1}\oplus V_1,$ and each of these spaces has dimension at most $m-2\leq n-3.$ But this restriction must still be faithful. By the $S_{n-2}$ part of the induction hypothesis, this must be the (c) rep for $S_{n-2}.$ This forces $\dim V_{-1}=2.$ The $(-1)$-eigenspaces of any two disjoint transpositions in $S_{n-2}$ is a one-dimensional subspace of $V_{-1},$ which is independent of the choice of transpositions. So actually the $(-1)$-eigenspaces of any two transpositions in $S_{n-2}$ intersect in a fixed one-dimensional subspace, which is therefore a reducing subspace.