Non-abelian $p$-group; abelian subgroups of index $p$

Solution 1:

Note that in fact any abelian subgroup of index $p$ must contain the center: for if $H$ is maximal and abelian, and $Z(G)$ is not contained in $H$, then $G=HZ(G)$, which would make $G$ abelian. Therefore, if $H$ and $K$ are both abelian of index $p$, then $Z(G)\subseteq H\cap K$. If you have proven that $H\cap K\subseteq Z(G)$, then in fact we have that $H\cap K=Z(G)$ for any two distinct abelian subgroups of index $p$.

(And indeed: if $H\neq K$ are both abelian of index $p$, then for every $g\in G$ there exists $h\in H$, $k\in K$ with $g=hk$; given $x\in H\cap K$ we have $gx = (hk)x = x(hk) = xg$, since $x$ commutes with everything in $K$ and with everything in $H$, so $H\cap K\subseteq Z(G)$).

Conversely, if $M$ is a subgroup of $G$ that contains $Z(G)$ and is of index $p$, then it must be abelian (since $M/Z(G)$ is of order $p$). Thus, every abelian subgroup of $G$ of index $p$ corresponds to a subgroup of $G/Z(G)$, and you are done.

For part (b), show that (a) implies that if $G$ has more than one abelian subgroup of index $p$, then $G$ is of class exactly $2$: by the argument in (a), you know that $Z(G)$ has index $p^2$ in $G$, but that means that $G/Z(G)$ is abelian, which means that $[G,G]\subseteq Z(G)$.

So if $G$ is of class $3$ and order $16$, then it has at most one abelian subgroup of order $8$. The center must be of order $2$: if the center were of order $4$, then $G/Z(G)$ would be of order $4$, hence abelian, so again we have that $G$ would be of class $2$. Thus, $G/Z(G)$ is of order $8$, and cannot be abelian. Therefore, $G/Z(G)$ is one of the two nonabelian groups of order $8$. If $G/Z(G)$ were quaternion, then it would have four cyclic subgroups of order $4$: each of them pull back to a subgroup $H$ of $G$ of order $8$, containing the center; and it is not hard to show that they are all abelian, contradicting the fact that $G$ has at most one abelian subgroup of order $8$. So $G/Z(G)$ must be dihedral. Verify that you get a cyclic subgroup of order $8$ in $G$ in this case.

Added. So, let us suppose that $G$ is of class $3$ and order $16$, and has a unique abelian subgroup of index $2$. We know $Z(G)$ is of order $2$, and that $G/Z(G)$ is dihedral of order $8$. Let $z\in G$ generate $Z(G)$, and let $x\in G$ map to a generator of the cyclic group of order $4$ in $G/Z(G)$. We know that $x$ has order either $4$ or $8$ in $G$. We aim to show that it has order $8$.

If $x$ has order $8$, we are done. Otherwise, $|x|=4$. Let $y\in G$ be such that $y$ maps to the generators of $G/Z(G)$ of order $2$, so that $yx = x^3y$ or $yx=x^3yz$. In the former case, we have $yx^2 = x^6y = x^2y$, so $x^2$ commutes with $x$, $y$, and $z$; in the latter case we would have $yx^2 = x^3yxz = x^6yz^2 = x^2y$, so again $x^2$ commutes with $x$, $y$, and $z$. Since $x$, $y$, and $z$ generate $G$, it follows that $x^2\in Z(G)$. But then the image of $x$ in $G/Z(G)$ would have order $2$, which is a contradiction. Therefore, $x$ cannot have order $4$, and so must have order $8$, as desired.