Groups of order $pq$ have a proper normal subgroup
I am doing the following exercise from [Birkhoff and MacLane, A survey of modern algebra]:
Let $G$ be a group of order $pq$ ($p,q$ primes). Show that either $G$ is cyclic or contains an element of order $p$ (or $q$). In the second case, show that $G$ contains either $1$ normal or $q$ conjugate subgroups of order $p$. In the latter case the $pq-(p-1)q=q$ elements not of order $p$ form a normal subgroup. Infer that $G$ always has a proper normal subgroup.
I think I was able to prove $G$ has a proper normal subgroup, but without quite establishing that the number of subgroups of order $p$ is 1 or $q$.
Here is my solution:
Suppose $G$ is cyclic and $G=\langle x \rangle$. Then, $\langle x^p \rangle$ has order $q$ and is normal, so we are done. So suppose $G$ is not cyclic. Let $x \in G-1$. By Lagrange's theorem, the order of $x$ is $p$, say. Let $H:=\langle x \rangle$. $G$ acts on all subgroups of order $p$ by conjugation. In this action, the stabilizer of $H$ is the normalizer $N_G(H)$. Since $N_G(H) \supseteq H$, $|N_G(H)|$ equals $p$ or $qp$. Hence the orbit containing $H$ has length $q$ or 1 (by the orbit-stabilizer lemma). If this value is 1, then $H$ is normal, so we are done. If this value is $q$, then there are $pq-(p-1)q=q$ remaining nonidentity elements in $G$ that are not in any conjugate of $H$.
Case 1: If any of these $q$ elements, say some $y \in G$, has order $p$, then $K:=\langle y \rangle$ is either normal (so we are done) or has $q$ conjugate subgroups (in which case the conjugates of $H$ and of $K$ together contain $q(p-1)+q(p-1)+1>qp$ elements, a contradiction).
Case 2: If none of the $q$ elements has order $p$, then they all have order 1 or $q$, and hence form a cyclic subgroup $K$ of order $q$. $K$ must be normal, since otherwise $g^{-1}Kg \ne K$ for some $g \in G$, whereas $K$ already exhausted all the elements of $G$ of order $q$.
Thus, $G$ has a normal subgroup of order $p$ or $q$.
While I have shown the group is not simple, I think I haven't shown yet that the number of subgroups of order $p$ is 1 or $q$. The proof only establishes that the number of distinct conjugates of $H$ is 1 or $q$. Any suggestions on how to prove (using elementary methods) that any other subgroup of order $p$ would have to be conjugate to $H$? In other words, I need to rule out Case 1 in the proof.
In you proof (assuming $G$ is not cyclic), you have get that there exists a normal subgroup $K$ of order $q$, say. Let $H$ be a subgroup of order $p$. If $H$ is normal in $G$, then $G$ will be cyclic. We get $H$ has $q$ conjugates. But now we get $q+q(p-1)=|G|$ elements in $K $ and the conjugate of $H$. Hence there is no other subgroup of order $p$ except the conjugate of $H$. (All the proof holds under the condition $p \ne q$)
- You say "If this value is $q$ then there are $pq-(p-1)q=q$ remaining nonidentity elements in $G$..." Not quite. There are $q$ remaining elements. One of the remaining elements is the identity $1$.
Here are a few details that are implicit in your argument (probably too easy to need to be spelled out, but, better safe than sorry):
First, if $H$ is any subgroup with $p$ elements, note that every nonidentity element of $H$ is a generator for $H$.
If $H_1$ and $H_2$ are any two subgroups with $p$ elements each, if $H_1 \cap H_2$ contains any nonidentity element, then that element in the intersection is a generator for both $H_1$ and $H_2$. So $H_1=H_2$. Therefore, if $H_1$ and $H_2$ are any distinct $p$-element subgroups, then they are also disjoint (except for the identity element).
This is the reason why the union of the $q$ conjugate groups of $H$ contains $q(p-1)$ nonidentity elements.
- You want to rule out Case 1. It seems to me that your argument does this. Can you say what part worries you? If there is an element $y$ which is not in any conjugate of $H$, and $y$ has order $p$, then $K = \langle y \rangle$ is another subgroup of order $p$, and the conjugates of $K$ again contain $q(p-1)$ nonidentity elements. No conjugate of $K$ shares any nonidentity element with any conjugate of $H$, otherwise $K$ and $H$ would be conjugate. So all together these conjugates of $K$ and $H$ have $2q(p-1)$ nonidentity elements, or $2q(p-1)+1$ elements in total. But $2q(p-1)+1=pq+q(p-2)+1>pq$, as you noted. This is impossible, the union of the conjugates can't have more elements than $G$. So the element $y$ can't have order $p$. This shows that every element $y$ outside the conjugates of $H$ has order $q$.
Since $q$ is prime, every nonidentity element $K = \langle y \rangle$ has order $q$. This accounts for all the $q-1$ elements of order $q$. So, this set of $q$ elements left over from the conjugates of $H$ must be equal to $K$.
And $K$ must be normal because any conjugate of $K$ consists of order $q$ elements, but all the order $q$ elements are already in $K$.
In conclusion, your argument seems fine (modulo a minor wording thing in point (1) above), and in particular it seems to me that you already ruled out Case 1.