Find subgroup $\langle a,b\rangle$ of $\Bbb Z_{20}^*$ which is not cyclic.

Solution 1:

Since you already know $G=\Bbb Z_{20}^*$ is not cyclic, and every group is a subgroup of itself, you could take the subgroup to be $\Bbb Z_{20}^*$.

Less glib, follow @MeanX's comment.

Since $3$ and $11$ in $G$, $3\times 11=11\times 3$, $3\notin\langle 11\rangle=\{1,11\}$, and $11\notin\langle 3\rangle =\{1,3,9,7\}$, we have

$$\begin{align} \langle 3,11\rangle &\cong \langle x,y\mid x^2, y^4, xy=yx\rangle \\ &\cong \Bbb Z_2\times\Bbb Z_4, \end{align}$$

which is not cyclic.

But $|\Bbb Z_2\times\Bbb Z_4|=8=\varphi(20)=|G|$.

A proper subgroup can be found similarly by considering $\langle 9, 11\rangle$.

That subgroup is isomorphic to $\Bbb Z_2\times\Bbb Z_2$, the Klein four group.

Solution 2:

Well, note that $H\doteq \langle 3,11\rangle$ has order $8$ as $3$ does not generate $11$ in $(\mathbb{Z}/20\mathbb{Z})^{\times}$, and $\langle 3 \rangle$ has order $4$, [as $3^4 = 81 \equiv_{20} 1$] while $11^4 = (121)^2 \equiv_{20} 1$. And so as $H$ is abelian, from this it follows that $h^4 \equiv_{20} 1$ for each $h \in H$. Thus, every element in $H$ generates at most $4 <|H|$ elements, so there is no element in $H$ that generates the entire group. Can you finish from here?

Fun fact: You do know that $H\doteq \langle 3,11\rangle$ is in fact all of $(\mathbb{Z}/20\mathbb{Z})^{\times}$, right?