What are the subgroups of $\operatorname{Alt}(2p)$ containing the normalizer of a Sylow $p$-subgroup?

Solution 1:

I think the maximality of $N$ in $M$ follows from the fact that a sharply 2-transitive Frobenius group $F_p$ of order $p(p-1)$ is maximal in $S_p$, which can be verified using the list of all 2-transitive permutation groups.

Let $N_0$ be the subgroup of index 2 in $N$ that fixes both orbits of $P$. Then the projection of $N_0$ onto each of these orbits is equal to $F_p$. For any group $L$ with $N < L \le M$, the projection of $L_0$ onto these orbits would be larger than $F_p$ and hence would equal $S_p$. But now, from the simplicity of $A_p$, we can deduce that $L_0=M_0 = (A_p \times A_p).2$, so $L=M$.

Solution 2:

The proof seems quite long, and requires the classification of finite simple groups. Perhaps the most surprising part to me is that the similar question in prime degree $p$ (rather than degree $2p$ as asked) still requires the classification to handle general primes, and requires quite a bit of modular representation theory to handle primes of special forms. At any rate, here is what I've had time to write up.

Itô's conjecture and Wielandt's problem

Is the normalizer $F$ of a Sylow $p$-subgroup of the symmetric group on p points maximal?

This seemed to be true. The first results along these lines were Galois's investigations on groups of prime degree, followed by Mathieu's discovery of his sporadic groups (especially of prime degree). Burnside used ordinary character theory to show that a transitive group of prime degree was either a subgroup of $F$ or insoluble. Brauer (1943) applied his ideas of modular character theory to give some bounds. Itô (1958+) began to get detailed information on groups of prime degree when the primes were of special form. Wielandt's 1964 textbook (Part V, especially §31) gathered together Burnside, Schur, and to some extent Brauer's work and gave representation theoretic proofs of several of these results. Neumann (1972) used modular representation theory to show that any overgroup is triply transitive (see Huppert–Blackburn XII.10.9). Chillag (1977) showed that if the overgroup inverted an element of order $p−1$ in the normalizer, then it was the entire symmetric group. Finally after the CFSG was declared complete, lists of $2$-transitive groups became available, a published list being clearly presented in Kantor (1985).

List of $2$-transitive permutation groups

Such groups divide into $2$-sorts by Galois: those with elementary abelian socle and those with simple socle.

The ones with simple socle $N$ are almost simple and have degree $v$ as in the following list:

• $v ≥ 5$, $N = \operatorname{Alt}(v)$
• $v = (q^d−1)/(q−1)$, $N = \operatorname{PSL}(d,q)$, $q$ a prime power, $(d,q) ≠ (2,2), (2,3)$, and two representations if $d ≥ 3$
• $v = q^3+1$, $N = \operatorname{PSU}(3,q)$, $q$ a prime power, $q > 2$
• $v = q^2+1$, $N = \operatorname{Sz}(q)$, $q=2^{2e+1}$, $e$ a positive integer
• $v = q^3+1$, $N = \operatorname{Ree}(q)$, $q=3^{2e+1}$, $e$ a positive integer
• $v = 2^{2n−1} ± 2^{n−1}$, $N = \operatorname{Sp}(2n,2)$, $n ≥ 3$
• $v = 11$, $N = \operatorname{PSL}(2,11)$, two representations
• $v \in \{ 11, 12, 22, 23, 24 \}$, $N = \operatorname{Mathieu}(v)$, and two representations for $v=12$
• $v = 12$, $N = \operatorname{Mathieu}(11)$
• $v = 15$, $N = \operatorname{Alt}(7)$, two representations
• $v = 176$, $N = \operatorname{HS}$, two representations
• $v = 276$, $N = \operatorname{Co}3$

The ones with abelian socle of order $v = p^d$ are all contained in $\operatorname{AΓL}(d,p)$. The subgroup $G_0$ of permutations fixing the $0$ vector comes from a relatively short list, and is a complement to the regular normal subgroup $V$ of order $v$. These should be on the list of transitive finite linear groups:

• $G ≤ \operatorname{AΓL}(1,v)$ is solvable, $G_0$ is cyclic of order dividing $v−1$
• $\operatorname{SL}(n,q) ⊴ G_0$, $q^n = p^d$
• $\operatorname{Sp}(n,q) ⊴ G_0$, $q^n = p^d$
• $G_2(q)' ⊴ G_0$, $q^6 = p^d$, $q$ even
• $G_0 \in \{ \operatorname{Alt}(6), \operatorname{Alt}(7) \}$, $v = 2^4$
• $\operatorname{SL}(2,q) ⊴ G_0$, $q \in \{3,5\}$, and either $d=2$, $p \in \{ 5, 7, 11, 19, 23, 29, \text{ or }59 \}$, or $d=4, p=3$.
• five more examples with $d=4, p=3$ ($G_0$ a semi-direct product of the quaternion-type extra-special group of order $32$ with one of the five transitive subgroups of $\operatorname{Sym}(5)$)
• $\operatorname{SL}(2,13) = G_0$, $d=6$, $p=3$.

Sifting the list (not done)

Now one has to check which of these groups actually have degree $2p$ (or just $p$). The second half of the list is easily dealt with: $2p$ is not a prime power unless it is $4$, and $p$ is exactly a prime, so the only possibility is $\operatorname{AΓL}(1,p) = \operatorname{AGL}(1,p)$, the normalizer of the Sylow $p$-subgroup.

The first half of the list is extremely difficult to analyze as it brings up questions similar to Fermat primes. However, we can also use that these groups have to contain the normalizer in $\operatorname{Alt}(2p)$ of a Sylow $p$-subgroup. There are six infinite families (alt, psl, unitary, suzuki, ree, symplectic) and a few sporadics.

The alternating group is not a maximal subgroup of itself.

In the case $2.\operatorname{PΓL}(d,q) = \operatorname{Aut}(\operatorname{PSL}(d,q))$, one would have $(q^d−1)/(q−1)=2p$. This number is even, so $q$ must be odd, and $d$ must be even. If $p$ divides $q−1$, then $p$ divides $|N|$ at least $d−1$ times. If $p$ divides the order of the socle $N$ only once, then $p$ must divide $|\operatorname{Out}(N)| = (q−1,d)⋅f⋅2$. If $p$ divides $f$, then $p$ divides $q−1$ by Fermat's little theorem, however, then $p$ divides $|N|$ at least $d−1$ times, so $d = 2$ and $2p = q+1$, so $q=2p-1$, but $q$ is also a $p$'th power. Since $q^p−2p+1$ is $0$ at $p=0$ and its derivative in $p$ is positive if $q ≥ 9$ (and since there are no small counterexamples), this is a contradiction. If $p$ divides $(q−1,d)$, then $d≥p$, and so $p$ divides $|N|$ at least $p−1$ times, which is too many since we can assume $p ≥ 3$. Hence $p$ divides $|N|$ exactly twice, and $[G:N]$ is coprime to $p$. If $p ≥ d$, then a Sylow $p$-subgroup of $N$ is abelian and semi-simple, so contained in a torus. In other words, the order $k$ of $q$ mod $p$ satisfies $d/3 < k ≤ d/2$. A permutation matrix effects the orbit swap, so we only need to see if all $(p−1)/2$ multiplication automorphisms from the normalizer of the Sylow $p$-subgroup in $\operatorname{Alt}(p)$ are contained in $G$. Hence we have to look at the normalizer of the torus…

The unitary and Ree cases are impossible as $q^3+1$ is divisible by $2$, $(q+1)/2$, and $qq-q+1$, so is not twice a prime. Suzuki has odd degree and sympletic cannot occur, $2$ divides $v$ only once, so $n=2$ is out of bounds.

Now the only sporadic possible is $M_{22}$, but its automorphism group is only divisible by $11$ once, so it cannot occur.

Bibliography

• Brauer (1943) MR0008237
• Ito (1960) MR0117283
  • and MR0124389 for larger index
  • and his 1963 MR0147535 where he raises the bar and begins the attack on his more general conjecture by proving multiple transitivity, which Neumann then simplifies and extends
  • and his 1967 MR0224697 where he handles Fermat primes
• Neumann (1972) MR0313369
  • and its application to Itô's conjecture MR0352227
• Chillag (1977) MR0437622
• Kantor (1985) MR0773556