Solution 1:

There're two principal axes in general, so

\begin{align*} \theta &=\frac{1}{2} \tan^{-1} \frac{B}{A-C}+\frac{n\pi}{2} \\ &= \tan^{-1} \left( \frac{C-A}{B} \color{red}{\pm} \frac{\sqrt{(A-C)^{2}+B^{2}}}{B} \: \right) \\ \end{align*}

The centre is given by $$(h,k)= \left( \frac{2CD-BE}{B^2-4AC}, \frac{2AE-BD}{B^2-4AC} \right)$$

Transforming to $$\frac{A+C \color{red}{\pm} \sqrt{(A-C)^{2}+B^{2}}}{2} x'^2+ \frac{A+C \color{red}{\mp} \sqrt{(A-C)^{2}+B^{2}}}{2} y'^2+ \frac {\det \begin{pmatrix} A & \frac{B}{2} & \frac{D}{2} \\ \frac{B}{2} & C & \frac{E}{2} \\ \frac{D}{2} & \frac{E}{2} & F \end{pmatrix}} {\det \begin{pmatrix} A & \frac{B}{2} \\ \frac{B}{2} & C \\ \end{pmatrix}}=0$$

where $\begin{pmatrix} x' \\ y' \end{pmatrix}= \begin{pmatrix} \cos \theta & \sin \theta \\ -\sin \theta & \cos \theta \end{pmatrix} \begin{pmatrix} x-h \\ y-k \end{pmatrix}$.

The axes will match, up to reflection about the axes of symmetry, when the $\color{red}{\text{case}}$ (upper or lower) agree.

Numerical example

Given five points: $(2,1)$, $(1,1)$, $(-2,-2)$, $(-1,-2)$, $(1,-1)$

$A=1$, $B=-2$, $C=2$, $D=-1$, $E=2$, $F=-2$

$$(h,k)=(0,-0.5)$$

$$\det \begin{pmatrix} A & \frac{B}{2} & \frac{D}{2} \\ \frac{B}{2} & C & \frac{E}{2} \\ \frac{D}{2} & \frac{E}{2} & F \end{pmatrix} = ACF-\frac{A E^2+C D^2+F B^2-EDB}{4}=-\frac{5}{2}$$

$$\det \begin{pmatrix} A & \frac{B}{2} \\ \frac{B}{2} & C \end{pmatrix} = -\frac{B^2}{4}+AC=1$$

$$\frac{A+C \pm \sqrt{(A-C)^{2}+B^{2}}}{2}=\frac{3 \pm \sqrt{5}}{2}$$

Using upper case convention:

$$\frac{3+\sqrt{5}}{2} x'^2+\frac{3-\sqrt{5}}{2} y'^2=\frac{5}{2}$$

$$\frac{x'^2}{a^2}+\frac{y'^2}{b^2}=1$$

$$(x',y')= (a\cos t,b\sin t)$$

where $\displaystyle \begin{pmatrix} a \\ b \end{pmatrix}= \begin{pmatrix} \sqrt{\frac{5}{3+\sqrt{5}}} \\ \sqrt{\frac{5}{3-\sqrt{5}}} \end{pmatrix}$

$$\theta = \tan^{-1} \left( \frac{C-A}{B}+\frac{\sqrt{(A-C)^{2}+B^{2}}}{B} \: \right) = \tan^{-1} \left( -\frac{\sqrt{5}+1}{2} \right) \approx -58.28^{\circ} $$

\begin{align*} \begin{pmatrix} x \\ y \end{pmatrix} &= \begin{pmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{pmatrix} \begin{pmatrix} x' \\ y' \end{pmatrix}+ \begin{pmatrix} h \\ k \end{pmatrix} \\ &&\\ &= \begin{pmatrix} h+x'\cos \theta-y'\sin \theta \\ k+x'\sin \theta+y'\cos \theta \end{pmatrix} \\ &&\\ &= \begin{pmatrix} \sqrt{\frac{5}{2}+\sqrt{5}\,} \, \sin t+ \sqrt{\frac{5}{2}-\sqrt{5}\,} \, \cos t \\ -\frac{1}{2}+ \frac{\sqrt{5+\sqrt{5}}}{2} \, \sin t- \frac{\sqrt{5-\sqrt{5}}}{2} \, \cos t \end{pmatrix} \end{align*}

Solution 2:

Justification of the formula:

After centering (translation to let the linear terms vanish), the equation becomes $$Ax^2+Bxy+Cy^2+F'=0.$$

Then you apply a rotation of angle $\theta$ to let the mixed term $Bxy$ vanish from the quadratic terms,

$$A(x\cos\theta-y\sin\theta)^2+B(x\cos\theta-y\sin\theta)(x\sin\theta+y\cos\theta)+C(x\sin\theta+y\cos\theta)^2.$$

This is achieved when

$$2(C-A)\cos\theta\sin\theta+2B(\cos^2\theta-\sin^2\theta)=(C-A)\sin(2\theta)+2B\cos(2\theta)=0.$$

(Then you can rewrite $\frac{x^2}{a^2}+\frac{y^2}{b^2}=1$.)

Hence,

$$\tan(2\theta)=\frac{2B}{A-C}.$$

The angle $\theta$ is undeterminate by a multiple of a quater turn, as the ellipse has two symmetry axis.

Solution 3:

Although it sounds like a question, for calculation did you use atan2 function or atan function? Quadrant placement is also important.