Canonical form of conic section
Solution 1:
You want to eliminate the term involving $xy$. The simplest way for this example is to notice that $x^2+2xy=(x+y)^2-y^2$. So we use new variables $X=x+y, Y=y$ or $x=X-Y, y=Y$. Making this substitution gives $X^2-3Y^2+X-5Y=0$, and so it is a hyperbola.