Finding the asymptotes of a general hyperbola
I'm looking to find the asymptotes of a general hyperbola in $Ax^2 + Bxy + Cy^2 + Dx + Ey + F = 0$ form, assuming I know the center of the hyperbola $(h, k)$. I came up with a solution, but it's too long for me to be confident that I didn't make a mistake somewhere, so I was wondering if I could run it by someone and see if it works. It's mostly algebraic, and I'm prone to making tiny errors in algebra that throw off the entire problem.
So to start, since we know the center $(h, k)$, we can first translate the hyperbola by $(-h, -k)$ using the transform $x_0 = x - \Delta{x}, y_0 = y - \Delta{y}$ with $\Delta{x} = -h$ and $\Delta{y} = -k$. Assuming $F'$ is the translated $F$, we can divide the entire equation by $-F'$ to put it in the following form:
$$ ax^2 + bxy + cy^2 + dx + ey = 1 $$ With $a = -A'/F'$ and $A'$ the translated $A$, $b = -B'/F'$ and $B'$ the translated $B$, and so on.
Next we convert to polar coordinates to get the following:
$$ r^2(a\cos^2{\theta} + b\cos{\theta}\sin{\theta} + c\sin^2{\theta}) + r(d\cos{\theta} + e\sin{\theta}) - 1 = 0 $$
Solving for $r$ will give us
$$ r = \frac{-d\cos{\theta} - e\sin{\theta} \pm \sqrt{(d\cos{\theta} + e\sin{\theta})^2 + 4(a\cos^2{\theta} + b\cos{\theta}\sin{\theta} + c\sin^2{\theta})}}{2(a\cos^2{\theta} + b\cos{\theta}\sin{\theta} + c\sin^2{\theta})} $$
Now assume $\theta_0 = 2(a\cos^2{\theta} + b\cos{\theta}\sin{\theta} + c\sin^2{\theta})$, this means that as $\theta_0 \rightarrow 0^{\pm}$, $r \rightarrow \pm\infty$. The angles at which $r \rightarrow \pm\infty$ are the asymptotes of the hyperbola, so now it's just a matter of solving for where $\theta_0 = 0$. This is where the majority of the algebra takes place and this is where I'm worried I made some miniscule mistake.
$$ \begin{align} & 2(a\cos^2{\theta} + b\cos{\theta}\sin{\theta} + c\sin^2{\theta}) = 0\\ & \Longleftrightarrow a(1 - \sin^2{\theta}) + b\cos{\theta}\sin{\theta} + c\sin^2{\theta} = 0 \\ & \Longleftrightarrow \sin^2{\theta}(c - a) + b\cos{\theta}\sin{\theta} = -a \\ & \Longleftrightarrow \frac{1 - \cos{2\theta}}{2}(c - a) + \frac{b}{2}\sin{2\theta} = -a \\ & \Longleftrightarrow (c - a)(1 - \cos{2\theta}) + b\sin{2\theta} = -2a \\ & \Longleftrightarrow (c - a)(1 - \cos{2\theta}) + 2a = -b\sqrt{1 - \cos^2{2\theta}} \\ & \Longleftrightarrow \frac{a - c}{b}(1 - \cos{2\theta}) - \frac{2a}{b} = \sqrt{1 - \cos^2{2\theta}} \\ & \Longleftrightarrow (\frac{a - c}{b}(1 - \cos{2\theta}))^2 - 4a\frac{a - c}{b^2}(1 - \cos{2\theta}) + (\frac{2a}{b})^2 = 1 - \cos^2{2\theta} \\ & \Longleftrightarrow (\frac{a - c}{b})^2(1 - 2\cos{2\theta} + \cos^2{2\theta}) - 4a\frac{a - c}{b^2}(1 - \cos{2\theta}) + (\frac{2a}{b})^2 = 1 - \cos^2{2\theta} \\ & \Longleftrightarrow [(\frac{a - c}{b})^2 + 1]\cos^2{2\theta} + [2(\frac{a - c}{b})(\frac{a}{b} - \frac{a - c}{b})]\cos{2\theta} + [(\frac{2a}{b})^2 + (\frac{a - c}{b})^2 - 4a\frac{a - b}{b^2}) - 1] = 0 \\ & \Longleftrightarrow [(\frac{a - c}{b})^2 + 1]\cos^2{2\theta} + 2(\frac{a - c}{b})(\frac{c}{b})\cos{2\theta} + [(\frac{2a}{b})^2 + (\frac{a - c}{b})(\frac{-3a - c}{b}) - 1] = 0 \\ & \Longleftrightarrow [(\frac{a - c}{b})^2 + 1]\cos^2{2\theta} + 2(\frac{a - c}{b})(\frac{c}{b})\cos{2\theta} + [\frac{a^2 + 2ac + c^2}{b^2} - 1] = 0 \\ \end{align} $$ Now let $$ U = [(\frac{a - c}{b})^2 + 1] \\ V = 2(\frac{a - c}{b})(\frac{c}{b}) \\ W = \frac{a^2 + 2ac + c^2}{b^2} - 1 $$ So that the above equation becomes $$ U\cos^2{2\theta} + V\cos{2\theta} + W = 0. $$ Solving for $\cos{2\theta}$ gives us $$ \cos{2\theta} = \frac{-V \pm \sqrt{V^2 - 4UW}}{2U}. $$ Finally we may solve for theta like so, $$ \theta = \frac{1}{2}\arccos{\frac{-V \pm \sqrt{V^2 - 4UW}}{2U}}. $$ This gives us two numbers, $\theta_1$ and $\theta_2$, each corresponding to the slopes $m_1$ and $m_2$ of the asymptotes.
The relationship between the slope of a line $m$ and the angle $\theta$ between the line and the positive x-axis is $m = \tan{\theta}$. You can use the identity $\tan{(\frac{1}{2}\arccos{x})} = \sqrt{\frac{1 - x}{x + 1}}$ to solve for $m_1$ and $m_2$ in terms of non-trig functions, but I think this answer is sufficient enough.
Now that we have the slopes of the asymptotes, we can find the y-intercepts $b_1$ and $b_2$ of each line by simply plugging in the original center $(h, k)$ into each equation for the line and solving.
$$ b_1 = k - m_1h \\ b_2 = k - m_2h $$
Thus, the asymptotes of the hyperbola with general equation $Ax^2 + Bxy + Cy^2 + Dx + Ey + F = 0$ have equations $y = m_1x + b_1$ and $y = m_2x + b2$.
Does this look correct? Also, did I overcomplicate things? Was there an easy solution all along that I was missing?
So a fairly simple calculus based solution arises from knowing that the Asymptotes are the points where the slope tends toward being constant
That is given
$$Ax^2 + By^2 + Cxy + Dx + Ey + F = 0$$
We begin by deriving with respect to $x$
$$2Ax + 2By \frac{dy}{dx} + Cy+Cx\frac{dy}{dx} + D + E\frac{dy}{dx} = 0 $$
And now solve for $\frac{dy}{dx}$
$$\frac{dy}{dx} = -\frac{2Ax + Cy + D}{2By + Cx + E} $$
Now we can take the limit as $x \rightarrow \pm \infty$ to determine the slopes of the Asymptotes and call this values $m$
Now take the original curve, and solve it for $y$ (it won't be too pretty)
$$y = \frac{-E- Cx \pm \sqrt{(E + Cx)^2 - 4B(Dx + F)}}{2B} $$
And evaluate $$ \lim_{x \rightarrow \pm \infty} y(x) - mx $$
to recover the y intercepts of the asymptotes.
This ENTIRE process can now be repeated with $\frac{dx}{dy}$ in place of $\frac{dy}{dx}$ and y exchanged with x to interpret the asymptotes in a reflected coordinate systems
For conics $$ax^2+2hxy+by^2+2gx+2fy+\color{blue}{c}=0$$
which is a hyperbola when $ab-h^2<0$.
Its asymptotes can be found by replacing $\color{blue}{c}$ by $\color{red}{c'}$ where
$$\det \begin{pmatrix} a & h & g \\ h & b & f \\ g & f & \color{red}{c'} \end{pmatrix} =0$$
That is
$$\color{red}{c'}=\frac{af^2+bg^2-2fgh}{ab-h^2}$$
The asymptotes are
$$\fbox{$ax^2+2hxy+by^2+2gx+2fy+\frac{af^2+bg^2-2fgh}{ab-h^2}=0 \,$}$$
Alternatively, using the centre of the conics
$$ \left( \frac{bg-fh}{h^2-ab}, \frac{af-gh}{h^2-ab} \right)$$
and the slope $m$ of an asymptote is given by
$$a+2hm+bm^2=0$$
On solving,
$$m=\frac{-h \pm \sqrt{h^2-ab}}{b}=\frac{a}{-h \mp \sqrt{h^2-ab}}$$
Therefore
$$\fbox{$ y-\frac{af-gh}{h^2-ab}= \frac{-h \pm \sqrt{h^2-ab}}{b} \left( x-\frac{bg-fh}{h^2-ab} \right) $}$$
See another answer here for your interest.