Union of the conjugates of a proper subgroup

Let G be a finite group and H be a proper subgroup. Prove that the union of the conjugates of H is not the whole of G.

Thanks for any help

(Note: Finite was not specified when I wrote this answer; I'll keep the more general answer, though)

The result is true if we assume that $H$ is of finite index. It may be false if $H$ is of infinite index.

For a counterexample in the infinite index case, let $F$ be an algebraically closed field, let $G$ be the group of all $n\times n$ invertible matrices with coefficients in $F$, and let $H$ be the subgroup of upper triangular matrices. Since every matrix over an algebraically closed field is similar to an upper triangular matrix (e.g., the Jordan canonical form), it follows that the union of conjugates of $H$ equals the whole group, even though $H$ does not equal all of $G$.

For a proof in the finite index case, let $[G:H]=n$. Then the action of $G$ on the cosets $H$ by left multiplication gives a homomorphism $G\to S_n$ with kernel $K\subseteq H$. This reduces to the finite case.

In the finite case, let $|H|=k$; then $|G|=kn$. There are at most $n$ distinct conjugates. Since the identity element is in all of the conjugates, the union of the conjugates of $H$ has at most $$n(k-1)+1 = nk-n+1\text{ element}$$ and since we are assuming $n\gt 1$, it follows that $$\left|\bigcup_{g\in G}gHg^{-1}\right| \leq nk-(n-1) \lt nk = |G|,$$ so the union cannot equal all of $G$.

I think the Orbit-Stabilizer Theorem can be applied here.

Let $G$ have order $n$, and since $H$ is a proper subgroup, let $[G\colon H]=m>1$. Let $N(H)$ be the normalizer of $H$ in $G$, which contains $H$. As such, $[G\colon N(H)]\leq[G\colon H]$.

Let $G$ act by conjugation, so that the orbit of $H$ is the set of all conjugate subgroups. So the stabilizer of $H$ is precisely the set $N(H)$, so by the Orbit-Stabilizer Theorem, the number of all conjugate subgroups is equal to $[G\colon N(H)]$. Now each of the conjugate subgroups has cardinality equal to that of $H$, and each contains the identity element $e$, so there are most $1+[G\colon N(H)](\vert H\vert-1)$ elements in the union. So $$ 1+[G\colon N(H)](\vert H\vert-1)\leq 1+[G\colon H](\vert H\vert-1)=1+\vert G\vert-m=\vert G\vert+(1-m)<\vert G\vert $$ since $m>1$. So the union of the conjugate subgroups is a proper subset.

I recently encountered a nice exercise in Isaacs, Finite Group Theory, which allows us to say more.

Theorem: if $G$ is a finite group and $H < G$ is a proper subgroup, then the number of elements of $G$ which do not lie in any conjugate of $H$ is at least $|H|$. Since we always have $|H| \geq 1$, this result implies the one in the OP.

The proof uses the permutation character, which is defined as follows. If $G$ acts on the set $\Omega$, then the permutation character is the integer-valued function $\chi$ which counts the number of elements of $\Omega$ fixed by each $g \in G$: $$\chi(g) = |\{\alpha \in \Omega : g \cdot \alpha = \alpha\}|$$

To obtain the desired result, we define an appropriate group action and apply [PC]. Let $\Omega$ be the set of all left cosets of $H$. Then $G$ and $H$ both act on $\Omega$ by left multiplication.

The action by $G$ is clearly transitive; there is only one orbit.

I will reserve $n$ to refer to the number of orbits under the action by $H$. Note that $n \geq 2$ because $\{H\}$ is one orbit, but it can't be the only orbit because $H$ is a proper subgroup, so there is more than one coset of $H$.

The permutation character for the action by $G$ is $$\chi(g) = |\{aH \in \Omega : gaH = aH\}|$$ and the permutation character for the action by $H$ is simply the restriction of $\chi$ to $H$.

Notice that $gaH = aH$ iff $ga \in aH$ iff $g \in aHa^{-1}$, so the stabilizer of $aH$ under the action by $G$ is $aHa^{-1}$, and under the action by $H$ it is $H\ \cap\ aHa^{-1}$. For the action by $H$, the identity [PC] becomes $$\sum_{h \in H}\chi(h) = \sum_{aH \in \Omega}|H\ \cap\ aHa^{-1}| = n|H| \geq 2|H|$$ For the action by $G$, [PC] gives us $$\begin{aligned} |G| &= \sum_{g \in G}\chi(g) \\ & = \sum_{h \in H}\chi(h) + \sum_{g \in G \setminus H} \chi(g) \\ &\geq 2|H| + \sum_{g \in G \setminus H} \chi(g) \end{aligned}$$ The number of terms in the rightmost sum is $|G| - |H|$. Note that $\chi(g) \geq 1$ iff $g$ fixes at least one element of $\Omega$ iff $g$ lies in some conjugate of $H$. If we define $Z$ to be the set of elements of $G$ which lie in no conjugate of $H$, then clearly $Z \subseteq G \setminus H$, so in the rightmost sum there are $|G| - |H|$ terms, of which $|G| - |H| - |Z|$ are nonzero. Therefore the rightmost sum is at least $|G| - |H| - |Z|$. This gives us $$|G| \geq 2|H| + |G| - |H| - |Z|$$ so $|Z| \geq |H|$, which proves the theorem.