The Ellipse Problem - finding an ellipse inside a triangle

The problem statement is as follows:

A triangle is dissected into six smaller triangles by its angle bisectors. Prove that the intersections of the angle bisectors of each of these smaller triangles lie on an ellipse.

Could anybody here solve it?

[Deleting previous answer, and starting over.]

This is a special case of a more general result (which I suspect must be known, but wasn't terribly difficult to derive independently) about when the endpoints of three concurrent segments lie on a conic (are "co-conicear"?).

Proposition. Suppose six distinct points, $P_0$, $P_1$, $P_2$, $Q_0$, $Q_1$, $Q_2$ are such that the lines $P_i Q_i$ are distinct and concur at a distinct seventh point, $R$. Write $p_i := \pm|RP_i|$, $q_i := \pm|RQ_i|$, and $\theta_i = \angle P_{i-1}RP_{i+1}$; for each $i$, we assign $p_i$ and $q_i$ opposite signs if $R$ is between $P_i$ and $Q_i$, and identical signs otherwise. (The $p_i$ and $q_i$ cannot be zero, nor equal.) Then, those six points lie on a common conic if and only if

$$ \sum_i \left(\frac{1}{p_i}+\frac{1}{q_i}\right)\sin\theta_i = 0$$

Proof. Take $R$ to be the origin, and assign coordinates to the points:

$$\begin{eqnarray*} P_0 = p_0 (1, 0) \hspace{0.25in} P_1 = p_1(\cos\phi_1, \sin\phi_1) \hspace{0.25in} P_2 = p_2 (\cos\phi_2,\sin\phi_2)\\ Q_0 = q_0 (1, 0) \hspace{0.25in} Q_1 = q_1(\cos\phi_1, \sin\phi_1) \hspace{0.25in} Q_2 = q_2 (\cos\phi_2,\sin\phi_2) \end{eqnarray*}$$

Substitute the coordinates into the conic template $A x^2 + 2 B xy + Cy^2 + 2 D x + 2 Ey + 1 = 0$ to get a system of six equations in five unknown coefficients. (Note that any such conic will not pass through the origin, or $R$ in the general context.) A computer algebra system helps to determine that the system imposes a dependency among the $p_i$, $q_i$, and $\theta_i$ ---for instance, evaluating the determinant in equation (8) of MathWorld's "Conic Section" entry, with the sixth point's coordinates entered into the top row--- that dependency can be expressed as the summation above, with (taking $\phi_0 := 0$) $\sin\theta_i := \sin(\phi_{i+1}-\phi_{i-1})$. Note that our assumptions provide that $p_i \ne 0$, $q_i \ne 0$, $p_i \ne q_i$, and that $\sin\theta_i\ne 0$.

To the problem at hand ...

The Proposition is clearly relevant, since pairs of a triangle's "sub-incenters" determine three segments that concur at the triangle's incenter. We just have to figure out the lengths and angles involved to run the "co-conicearity test".

First, some notation: Let $\triangle ABC$ have incenter $I$, and let $N_A$, $N_B$, $N_C$ be the points where angle bisectors $AI$, $BI$, and $CI$ meet sides $BC$, $CA$, and $AB$, respectively. (If you consult the first figure in MathWorld's "Incircle" entry, my $N$s are close to their $M$s ... which is why I chose "$N$" to denote them.) Let $P_{AB}$ be the incenter of $\triangle IAN_C$; the "$AB$" indicates that the triangle lies along $A$'s end of side $AB$. Likewise, let $P_{BC}$ and $P_{CA}$ be the incenters of $\triangle IBN_A$ and $\triangle ICN_B$; and let $Q_{BA}$, $Q_{CB}$, $Q_{AC}$ be the incenters of $\triangle IBN_C$, $\triangle ICN_A$, and $\triangle IAN_B$. (The concurrent segments are then $P_{CA}Q_{BA}$, $P_{AB}Q_{CB}$, $P_{BC}Q_{AC}$.)

Because $AI$, $BI$, and $CI$ bisect vertex angles, and each $P_\star I$ and $Q_\star I$ bisects an angle at $I$, finding (the sines of) the angles between segments is straightforward, as shown below.

Note. To save space, I write "$\theta_n$" for "$\frac{\theta}{n}$". Observe that $\alpha_n+\beta_n+\gamma_n = \pi_n$.

$$\begin{eqnarray*} \sin\angle P_{BC}IP_{CA} &=& \sin(\angle BIN_B - \angle BIP_{BC} - \angle N_B IP_{CA} \\ &=& \sin\left( \pi - \frac{1}{2} \angle BIN_A - \frac{1}{2}\angle CIN_B \right) \\ &=& \sin\left(\frac{1}{2}(\alpha_2+\beta_2)+\frac{1}{2}(\gamma_2+\beta_2)\right) = \sin\left(\alpha_4 + 2\beta_4+\gamma_4\right) \\ &=& \sin\left( \pi_4 + \beta_4 \right) \end{eqnarray*}$$

For the length information, we need to formulate the distance from a triangle's incenter to one of its vertices.

Let $\triangle ABC$ have edge lengths $a$, $b$, and $c$ (in the standard arrangement), and let $d$ be the triangle's circumdiameter (which of course, satisfies $d = \frac{a}{\sin\alpha} = \frac{b}{\sin\beta}=\frac{c}{\sin\gamma}$). Invoke the Law of Sines in $\triangle IBA$, and manipulate appropriately, to get the following

$$|IA| = \frac{|AB|\sin\angle IBA}{\sin(\pi-\angle IBA-\angle IAB)} = \frac{c\sin\beta_2}{\sin(\alpha_2+\beta_2)} = \frac{d\sin\gamma\sin\beta_2}{\cos\gamma_2}= 2 d \sin\beta_2\sin\gamma_2$$

(The use of the circumdiameter makes the symmetry clear.)

We want to apply that formula to, say, $\triangle IAN_C$, which has incenter $P_{AB}$. Within this triangle, the angles at $A$ and $N_C$ are, respectively $\alpha_2$ and (from looking at $ACN_C$) $\pi-\alpha-\gamma_2=\pi_2+\beta_2-\alpha_2$. The circumdiameter is

$$d_{AB} = \frac{|IA|}{\sin\angle N_C} = \frac{2d\sin\beta_2\sin\gamma_2}{\sin(\pi_2+\beta_2-\alpha_2)}=\frac{2d\sin\beta_2\sin\gamma_2}{\cos(\beta_2-\alpha_2)}$$

and we can compute

$$|IP_{AB}| = 2 d_{AB} \sin\frac{\alpha_2}{2}\sin\frac{\pi_2+\beta_2-\alpha_2}{2} = \frac{4d\sin\alpha_4 \sin\beta_2\sin\gamma_2\sin(\pi_4+\beta_4-\alpha_4)}{\cos(\beta_2-\alpha_2)}$$

That's a little ugly. If we conveniently scale $\triangle ABC$ so that $d\sin\alpha_2\sin\beta_2\sin\gamma_2=1$, then we have

$$\begin{eqnarray*} |IP_{AB}| &=& \frac{2\sin(\pi_4+\beta_4-\alpha_4)}{\cos\alpha_4\cos(\beta_2-\alpha_2)} = \frac{\sqrt{2}\left( \cos(\beta_4-\alpha_4)+\sin(\beta_4-\alpha_4)\right)}{\cos\alpha_4 \left( \cos^2(\beta_4-\alpha_4)-\sin^2(\beta_4-\alpha_4)\right)} \\ &=& \frac{\sqrt{2}}{\cos\alpha_4 \left(\cos(\beta_4-\alpha_4)-\sin(\beta_4-\alpha_4)\right)} = \frac{1}{\cos\alpha_4\cos\left(\pi_4+\beta_4-\alpha_4\right)} \end{eqnarray*}$$

Recall that the co-conicearity test requires reciprocating these lengths; convenient, indeed!

All we need to do now is compute the terms from the test's sum. With $I$ indisputably separating members of each $PQ$ pair, we know to assign opposite signs to the corresponding segment lengths, giving this factor

$$\begin{eqnarray*} \frac{1}{|IP_{AB}|}-\frac{1}{|IQ_{CB}|} &=& \cos\alpha_4 \cos(\pi_4+\beta_4-\alpha_4)-\cos\gamma_4\cos(\pi_4+\beta_4-\gamma_4) \\ &=& \frac{1}{2}\left( \cos(\pi_4-2\alpha_4+\beta_4)-\cos(\pi_4+\beta_4-2\gamma_4)\right) \\ &=& -\sin(\pi_4-\alpha_4+\beta_4-\gamma_4)\sin(\gamma_4-\alpha_4)\\ &=& -\sin\beta_2 \sin(\gamma_4-\alpha_4) \end{eqnarray*}$$

so that

$$\begin{eqnarray*} \left( \frac{1}{|IP_{AB}|}-\frac{1}{|IQ_{CB}|} \right) \sin\angle P_{BC}IP_{CA} &=& -\sin\beta_2 \sin(\gamma_4-\alpha_4)\sin(\pi_4+\beta_4) \\ &=& \frac{1}{2} \sin\beta_2 \left( \sin\gamma_2 - \sin\alpha_2 \right) \end{eqnarray*}$$

(The utter simplicity here makes me think that my approach has ignored a Putnamian insight.)

Clearly, the Proposition's cyclic sum vanishes, so that the six sub-incenters lie on a common conic.

Verification that the conic is, specifically, an ellipse is --for now-- left to the reader.