Suppose $G$ is a group generated by elements $x$ and $y$ where $xy^2 = y^3x$ and $yx^3 = x^2y$ What can you prove about $G$? [duplicate]

It's a pretty open-ended question, and therefore hard to answer. But, here are a couple of things to look at and how they apply in this example.

First, you should wonder whether the group is finite or not. If you've got more generators than relators (or relations) then your group is infinite. However, this does not apply here, since the numbers of generators and relations are both equal to two.

The next thing to look at is the abelianisation, $G/[G,G]$. For a finitely presented group, this is easy to do. First write your relations as relators: $$G = \langle x,y\mid x^{-1}y^{-3}xy^{2}, y^{-1}x^{-2}yx^{3}\rangle .$$ Now compute the relation matrix for the presentation. This is the matrix $(\sigma_{t}(w))$ of exponent sums of generators $t$ in each relator $w$ in the presentation. In this case, taking $w = x^{-1}y^{-3}xy^{2}$, for example, the exponent sum $\sigma_{x}(w)$ of $x$ in $w$ is $0$. (Just add up the exponents that appear on the letter $x$.) Similarly, $\sigma_{y}(w) = -1$. For the relator $w_{2} = y^{-1}x^{-2}yx^{3}$ we get $\sigma_{x}(w_{2}) = 1$ and $\sigma_{y}(w_{2}) = 0$. Therefore, the relator matrix is $$\begin{bmatrix} -1&0\\ 0&1\end{bmatrix}.$$ In general, you'd need to compute the Smith form for this to get the abelianised invariants, but in this case, there is nothing really to do, since it indicates that the abelianisation of $G$ is trivial. Therefore, you can conclude that $G$ is a perfect group (i.e., that $G = [G,G]$).

However, as soon as I see a finitely presented perfect group, I'm immediately suspicious that the group is itself trivial. (This need not be true, of course, but it's something to look into.) Therefore, you might try to show that the relations imply that the group is trivial.

Now, since it's me, I'd just go and ask a computer algebra system (such as Maple) to tell me.

> with(GroupTheory):                              
> G := < x,y | x.y^2 = y^3 . x, y.x^3 = x^2 . y >;
G :=

    < (x, y) |  (x^(-3).y^(-1).x^2.y, y^(-2).x^(-1).y^3.x)>

> GroupOrder( G );
                         1

But you can try to work it out by hand also, and for this example, it's not too difficult. (In general, it can be very difficult!) From the relation $xy^2 = y^3x$ we get $xy = y^3xy^{-1}$, and from $yx^3 = x^2y$ we get $xy = x^{-1}yx^3$. Hence, $y^3xy^{-1} = x^{-1}yx^3$. From this last equality we get $xy^3x = yx^3y$, from which we obtain $x^2yx = x^2y^2$ by using the original relations twice. Now this implies that $x = y$ by cancelling $x^2y$ on the left. Putting this back into the original relation $xy^2 = y^3x$, we get $x^3 = x^4$, whence, $x = 1$. Therefore, $y = 1$, and the group is trivial.

[Full disclaimer: It helped a lot that I knew the answer before I started.]