Rotating a conic section to eliminate the $xy$ term

define new variables $$x = x_1\cos t - y_1 \sin t , y = x_1 \sin t + y_1 \cos t$$ substituting in $5(x^2+y^2) - 6xy - 8= 0$ you get $$5(x_1^2 + y_1^2) - 6(x_1\cos t - y_1\sin t)(x_1\sin t + y_1 \cos t) - 8 = 0$$ which simplifies to $$x_1^2(5 - 6 \sin t \cos t) + y_1^2(5 +6\sin t \cos t)- 6x_1y_1(\cos^2 t - \sin ^2 t) - 8 = 0$$

you can eliminate the $x_1y_1$ term if you choose $t = 45^\circ$ which reduces the equation to the ellipse $$x_1^2 + 4y_1^2 = 4.$$

and if you choose $t = 135^\circ$ which reduces the equation to the ellipse $$4x_1^2 + y_1^2 = 4.$$

Did you try a less clever method?

Put $x=\alpha x' + \beta y'$ and $y=-\beta x' + \alpha y'$

Then the equation

$5x^2 + 5y^2 - 6xy - 8 = 0$ becomes

$(5(\alpha^2+\beta^2)+6\alpha\beta)x'^2+(5(\alpha^2+\beta^2)-6\alpha\beta)y'^2-6x'y'(\alpha^2-\beta^2)-8=0$

If you want to remove any cross product, then $\alpha^2-\beta^2=0$

EDITED after abel comment (forgotten contributing $x'^2$ and $y'^2$ coming from the $xy$ term).

It is easy to finish from here...