Proving the given identity in group

I think this nontrivial, and I doubt that this is the best way. But this does seem to work. My philosophy is that working with conjugates is easier and lets us systematically build a set of relations, so first rewrite the relations as a conjugate relation. Then work with it until we can substitute in the other relation. And hope.

As the two relations are the same, let's start with the first you gave. Start with $xy^2 = y^3x$. Rewriting it, we see that $xy^2x^{-1} = y^3$. This is a hopeful start. This says that $y^2$ is conjugate to $y^3$. This will be useful later.

Squaring, we see that $xy^4x^{-1} = y^6$. What does this do for us? Well... nothing. I want to keep going until we can make $x^3 y$ appear - that is the goal. So let's cube the conjugacy relation instead, yielding $xy^6x^{-1} = y^9$. While we're at it, let's take the fourth power as well, giving $xy^8x^{-1} = y^{12}$.

Aha! We have an expression for $y^6$ already from squaring the conjugacy relation. Substitution that into the cubic power, we get

$$x^2 y^4 x^{-2} = y^9.$$

I call this progress. We need this to happen again. So we square this last expression and get $x^2 y^8 x^{-2} = y^{18}$. What do we have now? Again... nothing. So we repeat what we've done before, and find the cube, $x^2 y^{12}x^{-2} = y^{27}$. Aha, we have a relation for $y^{12}$ from taking the fourth power above! Substituting that in here, we get

$$ x^3 y^8 x^{-3} = y^{27}.$$

We have finally put ourselves in a position where we can use the other relation. We know that $x^3y = yx^2$ (and so we also know that $y^{-1}x^{-3} = x^{-2}y^{-1}$, which we'll use as we're sticking with conjugates whenever possible). So we rewrite

$$\begin{align} x^3 y^8 x^{-3} &= y^{27} \\ (x^3y)y^8(y^{-1}x^{-3} &= y^{27} \\ y(x^2y^8x^{-2})y^{-1} &= y^{27} \end{align}$$

and because the fates like us, we already know that $x^2y^8x^{-2} = y^{18}$ from above. Using this, we substitute and get

$$y^{18} = y^{27}.$$

We have determined that $y^9 = e$. So in particular, the order of $y$ divides $9$. But from the first conjugacy relation $xy^2x^{-1} = y^3$, we see that $y^2$ and $y^3$ are conjugates, and thus have the same order. But as $2$ does not divide $9$, we see that the order of $y$ must actually by $1$, and so $y = e$. And thus $x = e$ as well.

Lemma 1: $y = x^{3k} y x^{-2k}$ (Induction using $y = x^3 y x^{-2}$)

Lemma 2: $y^n = x^{3^n} y^n x^{-2^n}$ (Induction using Lemma 1)

Lemma 3: $y^n = x^{3^n k} y^n x^{-2^n k}$ (Analogous to Lemma 1)

From Lemma 3 and $y^3 = x y^2 x^{-1}$, we get $$x^{27m} y^3 x^{-8m} = x^{9n + 1} y^2 x^{-4n - 1}$$ By substituting $y^3 = x y^2 x^{-1}$ we get $$x^{27m + 1} y^2 x^{-8m - 1} = x^{9n + 1} y^2 x^{-4n - 1}$$ By choosing $m = 1, n = 3$, we get $$x^{28} y^2 x^{-9} = x^{28} y^2 x^{-13}$$ $$x^4 = e$$

Thus, the order of $x$ divides 4. Since $x^3$ and $x^2$ are conjugates, they must have the same order. But then $x$ must be of order 1, i.e., $x = e$. And thus $y = e$ as well.