Problem from Herstein on group theory

• Suppose that $x\in G$ satisfies $ x^3=e$. Then we cannot have $x\ne e$, otherwise $x$ would have order 3, which implies that 3 divides $|G|$ (recall that the order of a group element divides the order of the group). Hence, $$\forall x\in G:\quad x^3=e~\Longrightarrow~x=e,$$ and as $(ab)^3=a^3b^3$ for all $ a,b \in G $, we see that the function $\phi:G\to G$ defined by $$\forall x\in G:\quad\phi(x) \stackrel{\text{def}}{=} x^3$$ is an injective group homomorphism.

• Now, $$\forall a,b\in G:\quad ababab=(ab)^3=a^3b^3=aaabbb.$$ Hence, $$\forall a,b\in G:\quad baba=aabb,\quad\text{or equivalently},\quad (ba)^2=a^2b^2.$$ Using this fact, we obtain \begin{align} \forall a,b\in G:\quad(ab)^4&= [(ab)^2]^2\\ &=[b^2a^2]^2\\ &=(a^2)^2(b^2)^2\\ &=a^4b^4\\ &=aaaabbbb. \end{align}

• On the other hand, \begin{align} \forall a,b\in G:\quad(ab)^4&= abababab\\ &=a (ba)^3b\\ &=ab^3a^3b\\ &=abbbaaab. \end{align}

• Hence, for all $a,b\in G$, we have $aaaabbbb=abbbaaab$, which yields $$\phi(ab)=a^3b^3=b^3a^3=\phi(ba).$$ As $\phi$ is injective, we conclude that $ab=ba$ for all $a,b\in G$.

Conclusion: $G$ is an abelian group.

The way, I am writing here, is from my old notes and personally I prefer the other approaches. But, maybe the given additional points below, inspire you for other problems like this problem.

We can prove that if for an integer $n$ and every $a,b\in G$, $(ab)^n=a^nb^n$, then $$(aba^{-1}b^{-1})^{n(n-1)}=e$$ The proof is easy. In fact, $$(aba^{-1}b^{-1})^{n^2}=[(aba^{-1}b^{-1})^n]^n=[a^n(ba^{-1}b^{-1})^n]^n=\cdots^*=a^nb^na^{-n}b^{-n}\\\ (aba^{-1}b^{-1})^{n}=(ab)^n(a^{-1}b^{-1})^n=a^nb^na^{-n}b^{-n}$$ ^{*refer here for the missing steps}

In your problem, we assume $G$ is not abelian, so there exist $a,b\in G, aba^{-1}b^{-1}\neq e$. According to the above lemma $$(aba^{-1}b^{-1})^6=e$$ since we know $(ab)^3=a^3b^3$. So $|aba^{-1}b^{-1}|\big| 6$ and because of $3\nmid|G|$ so $|aba^{-1}b^{-1}|=2$. This means that $(aba^{-1}b^{-1})^2=e$. On the other hand, $$(ab)^3=a^3b^3\Longrightarrow (ba)^2=a^2b^2$$ (see @Haskell's answer) then $(a^{-1}b^{-1})^2(ab)^2=e$ or $(ab)^2=(ba)^2=a^2b^2$ or $ab=ba$. A nice contradiction!

Hints (remember: $\,|G|<\infty\,\,\,and\,\,\,3\,\nmid\, |G|\,$): $\,\,\forall\,\,a,b\in G\,$

$$\begin{align*}(1)&\;\;\;\text{Show that}\,\,\,\,(ba)^2=a^2b^2\\{}\\(2)&\;\;\;\text{Prove that}\;\;f:G\to G\,\,\,,\,\,f(x):=x^3\,\,,\,\,\text{is an isomorphism}\\{}\\(3)&\;\;\;\text{Define}\,\,z:=\left(aba^{-1}\right)^3 \longrightarrow \begin{cases}z=ab^3a^{-1},\;\;\;\text{and also}\\{}\\z=f(a)f(b)f(a^{-1})=a^3b^3a^{-3}\end{cases}\\{}\\(4)&\;\;\;\text{Using(2)-(3) , show that}\;\;a^2\in Z(G)\Longleftrightarrow a^2g=ga^2\,\,,\,\forall\,g\in G\\{}\\(5)&\;\;\;\text{Finally, use (1) to show that}\,\,\,ab=ba\end{align*}$$