Finally, I am able to complete my sketch of the proof. We begin by proving the following:

$G := \langle x,y \mid x^3=y^3=(xy)^2 = 1 \rangle$ is isomorphic to $A_4$

Proof: $A_4$ is generated by $(123)$ and $(234)$, and these permutations satisfy the above relations. Hence, $A_4$ is a homomorphic image of $G$. We will show henceforth $|G| \leq 12$. Let $a = xy$ and $b = a^x = yx$. We have $a^2 = b^2 = 1$, and also $(ab)^2 = xy^{-1}x^{-1}y^{-1}x = x (xy)^{-2}x^2 = 1$. So $V := \langle a,b \rangle$ is a homomorphic image of $C_2 \times C_2$. Since $a^x = b \in V$ and $b^x = x^{-1}yx^2 = (yx)^{-1}(xy)^{-1} = ba \in V$, $\langle x \rangle$ normalizes $V$, and $G = V \langle x \rangle$ has at most 12 elements. $\square$

Now we are able to prove the original statement:

$G := \langle x,y \mid x^5=y^2=(xy)^3=1 \rangle$ is isomorphic to $A_5$

Proof: $A_5$ is generated by $(12345)$ and $(12)(34)$, and these permutations satisfy the above relations. Hence, $A_5$ is a homomorphic image of $G$. We will show $|G| \leq 60$. Let $a = xy$ and $b = a^{x^2} = x^{-1}yx^2$. We have $a^3=b^3=1$. In the following we will frequently need the identity

$$yx^{-1}y= xyx,\tag{$\ast$}$$

which follows directly from $(xy)^3=1$. Using $(*)$ we compute $(ab)^2 = x(yx^{-1}y)x^3(yx^{-1}y)x^2 = x(xyx)x^3(xyx)x^2 = 1$. Hence, $U := \langle a,b \rangle$ is a homomorphic image of $A_4$, and has therefore at most $12$ elements. We finish the proof by showing that the complete set of right cosets of $U$ in $G$ is given by $\Omega = \{ U, Ux, Ux^2, Ux^3, Ux^4 \}$. Since $G$ acts transitively on its right cosets, this can be done by showing that $\Omega$ is invariant under the action of the generators $x$ and $y$. It is clear that $\Omega x = \Omega$. Furthermore, we have

  • $Uy = Uay = Ux$
  • $(Ux)y = Ua = U$
  • $(Ux^2)y = Ux(xyx)x^{-1} = Ux(yx^{-1}y)x^{-1} = Uabx^2 = Ux^2$
  • $(Ux^3)y = Ub^{-1}x^4 = Ux^4$
  • $(Ux^4)y = Ubx^3 = Ux^3$

This also shows $\Omega y = \Omega$, and hence $\Omega G = \Omega$, which completes the proof. $\square$


I am quite satisfied with this proof, since it is very conceptual. But still it is quite long and depends on many calculations which seem a bit random. I would be happy to see shorter proofs which are using more sophisticated concepts.