Conic by three points and two tangent lines

The comment by Narasimham made me aware of a very elegant way of tackling this problem. The figure above can be interpreted as the orthogonal projection of a right cone whose axis lies in the plane. The cone intersects the plane in the two lines, $g$ and $h$. The points $A,B,C$ are in fact points on the cone, so they lie either above the plane or below the plane but are projected orthogonally into the plane. These three points in space define a plane, and that plane intersects the cone in a conic section. The orthogonal projection of that conic section is again a conic section, namely one of the four indicated in the figure. The four different solutions come from different choices about which of the points $A,B,C$ lie above the plane and which below. Since reflecting everything in the plane doesn't affect the resulting projected conic, one of the three points can be chosen arbitrarily, while the other two each allow for two possible choices, leading to $2^2=4$ generally distinct solutions.

So let's make this a bit more explicit. Using a suitable projective transformation defined by four points and their images, one can achieve a situation where the lines $g$ and $h$ intersect in the point $(0:0:1)$, the line $g$ intersects the line $AB$ in $(1:0:0)$ and the line $h$ intersects the line $AB$ in $(0:1:0)$. Furthermore, $C=(1:1:1)$ can be the fourth point defining this transformation. Then $A=(a:1:0)$ and $B=(b:1:0)$ describe the situation up to that projective transformation, so we only have to deal with two parameters $a,b\in\mathbb R$ except for some degenerate situations (like when $A$ or $B$ lies on $h$).

Now lift everything up to the cone. That cone has an aperture of $\frac\pi2$. In affine coordinates, you can describe it as the set of points $(x,y,z)$ which satisfies $(x + y)^2 = x^2 + y^2 + z^2$ or in other words $2xy = z^2$. But we are free to scale the $z$ coordinate by $\sqrt2$ so we might as well use

$$xy=z^2\tag1$$

as the equation of the cone. That equation is already homogeneous, so we can plug coordinates $(x:y:z:w)$ into that and find that $w$ is irrelevant. Translating out 2d points above to 3d we obtain $A=(a:1:\pm\sqrt a:0)$ and $B=(b:1:\pm\sqrt b:0)$ as well as $C=(1:1:1:1)$. The plane spanned by these three points is characterized by

$$\begin{vmatrix}1&\pm\sqrt a&0\\1&\pm\sqrt b&0\\1&1&1\end{vmatrix}x -\begin{vmatrix}a&\pm\sqrt a&0\\b&\pm\sqrt b&0\\1&1&1\end{vmatrix}y +\begin{vmatrix}a&1&0\\b&1&0\\1&1&1\end{vmatrix}z -\begin{vmatrix}a&1&\pm\sqrt a\\b&1&\pm\sqrt b\\1&1&1\end{vmatrix}w =0\tag2$$

If we introduce new symbols $p_i$ for the coefficients of this plane, we can shorten this to

\begin{align*} p_1x + p_2y + p_3z + p_4w &= 0 \\ p_1x + p_2y + p_4w &= -p_3z \\ (p_1x + p_2y + p_4w)^2 &= p_3^2xy \tag3 \end{align*}

This is a homogeneous quadratic equation in $(x:y:w)$ and as such describes a conic in the original plane. Now one might want to undo the projective transformation which led to the special coordinates, and then we are done. The four possible choices for the signs of $\pm\sqrt a$ and $\pm\sqrt b$ will lead to the four possible conics.