Finding Eccentricity from the rotating ellipse formula
Solution 1:
As the translation does not change the eccentricity of a conic, let's us define $x=X+h,y=Y+k$ to get $AX^2+BXY+CY^2=1$
Using Rotation of axes as eccentricity is one of the invariants in rotation,
$$x'^2(A\cos^2t+B\sin t\cos t+C\sin^2t)+x'y'(B\cos2t-(A-C)\sin2t)+y'^2(A\sin^2t-B\sin t\cos t+C\cos^2t)=1$$
To remove the $x'y'$ term, $\tan 2t=\frac{B}{A-C}\implies \frac{\sin2t}{B}=\frac{\cos2t}{A-C}=\pm\frac1{\sqrt{B^2+(A-C)^2}}$
If we take the $'+'$ sign, $\sin2t=\frac B{\sqrt{B^2+(A-C)^2}},\cos2t=\frac {A-C}{\sqrt{B^2+(A-C)^2}}$
Comparing with the standard form we get
$a^2=\frac1{A\cos^2t+B\sin t\cos t+C\sin^2t}$ $=\frac2{A(1+\cos2t)+B\sin2t+C(1-\cos2t)}$ $=\frac{2\sqrt{B^2+(A-C)^2}}{(A+C)\sqrt{B^2+(A-C)^2}+(A-C)^2+B^2}$ $=\frac2{A+C+\sqrt{B^2+(A-C)^2}}$
Similarly, $b^2=\frac2{A+C-\sqrt{B^2+(A-C)^2}}>a^2$
So, $b$ is the semi-major axis, $a$ is the semi-minor axis.
Consequently, the equation of the conic becomes $$\frac{x'^2}{\frac2{A+C+\sqrt{B^2+(A-C)^2}}}+\frac{y'^2}{\frac2{A+C-\sqrt{B^2+(A-C)^2}}}=1$$
$e^2=1-\frac{a^2}{b^2}=1-\frac{A+C-\sqrt{B^2+(A-C)^2}}{A+C+\sqrt{B^2+(A-C)^2}}=\frac{2\sqrt{B^2+(A-C)^2}}{A+C+\sqrt{B^2+(A-C)^2}}$
If $A+C-\sqrt{B^2+(A-C)^2}=0$ i.e., $4AC=B^2,e^2=1,$ the conic becomes a Parabola.
If $A+C-\sqrt{B^2+(A-C)^2}>0$ i.e., $4AC>B^2,e^2<1,$ the conic becomes an ellipse.
If $A+C-\sqrt{B^2+(A-C)^2}<0$ i.e., $4AC<B^2,e^2>1,$ the conic becomes a hyperbola.
Solution 2:
You find the eigenvalues of the matrix $M$ in Robert's answer here, $$M = \begin{bmatrix}A & B/2 \\ B/2 & C\end{bmatrix}.$$
The two eigenvalues of $M$ have an explicit formula since $M$ is a $2\times2$ matrix: $$\lambda = \frac12\left(A+C\pm\sqrt{(A-C)^2+B^2}\right).$$
Knowing these are equal to $1/a^2$ and $1/b^2$, you can find $a$ and $b$ to plug into the eccentricity formula.