Prove that the area of $\triangle DEF$ is twice the area of $\triangle ABC$

It is very interesting to solve this problem by only using our greek friends Pythagoras and Menelaus, together with some simple algebra.

Initial Observations

Without loss of generality assume that $P$ lies on the half-plane determined by $BC$ and not containing $A$. Note that there is no need to use the circle, once we fix $\angle BPC = 120°$. In the figure above I added point $G$ given by the intersection of line $FD$ with $AC$. Suppose also $\overline{AB}=1$.

Once $\overline{BF} = x$ is chosen, the entire Figure is defined. We aim therefore at showing that, independently of $x$, $$[DEF] = 2[ABC].$$

Characterization of $\triangle BFC$

Below the triangle in question has been isolated, to better help you in the demontrations.

Recall that $\overline{BC} = 1$ and $\overline{BF} = x$. Use Pythagorean Theorem on $\triangle CKF$ in oder to determine $$\overline{CF} = \sqrt{x^2+x+1}.$$ From the similarity $$\triangle BPC \sim \triangle BCF,$$ show that $$\overline{CF}\cdot\overline{CP} = 1,$$ so that, in the end, you have $$ \begin{cases} \overline{CF} = \sqrt{x^2+x+1},\\ \overline{CP} = \frac{1}{\sqrt{x^2+x+1}},\\ \overline{FP} = \frac{x(x+1)}{\sqrt{x^2+x+1}}. \end{cases} $$

Four Applications of Menelaus's Theorem (MT)

MT on $\triangle ACF$ with the line $BE$, gives $$\frac{\overline{CE}}{\overline{AE}} = \frac{1}{x+1}.$$ Together with $\overline{AE}-\overline{CE} = 1$ this leads to $$\overline{CE} = \frac{1}{x}$$ and $$\overline{AE} = \frac{x+1}{x}.$$

Similarly, MT on $\triangle BFC$ and line $AP$, with the known fact that $\overline{BD} + \overline{CD} = 1$ yields $$\overline{CD} = \frac{1}{x+1}$$ and $$\overline{DB} = \frac{x}{x+1}.$$ MT on $\triangle ABC$ and line $FG$, with $\overline{AG} + \overline{CG} = 1$, gives you $$ \overline{AG} = \frac{x+1}{x+2}$$ and $$ \overline{CG} = \frac{1}{x+2}.$$ Finally, MT on $\triangle CGF$ with line $AP$ yields $$ \frac{\overline{GD}}{\overline{DF}} = \frac{1}{x(x+2)}.$$

Areas Computation

Now we only need to compute areas of triangles with fixed altitude and a given ratio between bases.

Firstly we have $$[ACF] = [ABC](1+x).$$

Then $$[AFE] = [ACF] \frac{\overline{AE}}{\overline{AC}},$$ that is $$[AFE] = [ABC]\frac{(x+1)^2}{x}.$$

We also have $$[GFE] = [AFE]\frac{\overline{GE}}{\overline{AE}},$$ yielding $$ [GFE] =2[ABC]\frac{(x+1)^2}{x(x+2)}.$$

Finally observe that $$\frac{[DFE]}{[GDE]} = \frac{\overline{GD}}{\overline{DF}} = \frac{1}{x(x+2)}.$$

Thus we have the system of equations $$ \begin{cases} \frac{[DFE]}{[GDE]} = \frac{1}{x(x+2)}\\ [DFE] + [GDE] =2[ABC]\frac{(x+1)^2}{x(x+2)}. \end{cases} $$

leading, once solved, to the desired result, i.e. $$\boxed{[DFE] = 2[ABC]} $$

$\blacksquare$

Proof (for the case when $P$ lies on arc $BC$):

Let $AB=BC=CA=1$ and $\angle PAC=\theta$.

$$\frac{PD}{\sin\angle PBD}=\frac{PB}{\sin \angle PDB} \implies \frac{PD}{\sin\theta}=\frac{PB}{\sin (120^\circ-\theta)} $$

$$\frac{PB}{\sin\angle PAB}=\frac{AB}{\sin \angle APB}\implies \frac{PB}{\sin(60^\circ-\theta)}=\frac{AB}{\sin 60^\circ}$$

So, $\displaystyle PD=\frac{\sin\theta\sin(60^\circ-\theta)}{\sin60^\circ\sin(120^\circ-\theta)}=\frac{\sin\theta\sin(60^\circ-\theta)}{\sin60^\circ\sin(60^\circ+\theta)}$.

$$\frac{PE}{\sin\angle PAC}=\frac{AE}{\sin \angle APE} \implies \frac{PE}{\sin\theta}=\frac{AE}{\sin 120^\circ} $$

$$ \frac{AE}{\sin\angle ABE}=\frac{AB}{\sin\angle AEB} \implies \frac{AE}{\sin (60^\circ+\theta)}=\frac{AB}{\sin(60^\circ-\theta)}$$

So, $\displaystyle PE=\frac{\sin\theta\sin(60^\circ+\theta)}{\sin120^\circ\sin(60^\circ-\theta)}=\frac{\sin\theta\sin(60^\circ+\theta)}{\sin60^\circ\sin(60^\circ-\theta)}$.

$$\frac{PF}{\sin\angle PBF}=\frac{PB}{\sin \angle PFB} \implies \frac{PF}{\sin(120^\circ-\theta)}=\frac{PB}{\sin \theta}$$

So, $\displaystyle PF=\frac{\sin(120^\circ-\theta)\sin(60^\circ-\theta)}{\sin60^\circ\sin\theta}=\frac{\sin(60^\circ+\theta)\sin(60^\circ-\theta)}{\sin60^\circ\sin\theta}$.

The area of $\triangle DEF$ is

\begin{align*} &\;\frac{1}{2}(PD\cdot PE+ PE\cdot PF+PF\cdot PD)\sin 120^\circ\\ =&\;\frac{\sqrt{3}}{4}\left[\frac{\sin^2\theta+\sin^2(60^\circ+\theta)++\sin^2(60^\circ-\theta)}{\sin^260^\circ}\right]\\ =&\;\frac{1}{\sqrt{3}}\left[\sin^2\theta+(\sin60^\circ\cos\theta+\cos60^\circ\sin\theta)^2+(\sin60^\circ\cos\theta-\cos60^\circ\sin\theta)^2\right]\\ =&\;\frac{1}{\sqrt{3}}\left[\sin^2\theta+2\sin^260^\circ\cos^2\theta+\cos^260^\circ\sin^2\theta\right]\\ =&\;\frac{1}{\sqrt{3}}\left[\frac{3}{2}\sin^2\theta+\frac{3}{2}\cos^2\theta\right]\\ =&\;\frac{\sqrt{3}}{2} \end{align*}

The area of triangle $\triangle ABC$ is

$$\frac{1}{2}(1)^2\sin60^\circ=\frac{\sqrt{3}}{4}$$

Without loss of generality, let

\begin{align} O&=(0,0),\quad A=(1,0),\quad B=(-\tfrac12,\tfrac{\sqrt3}2),\quad C=(-\tfrac12,-\tfrac{\sqrt3}2),\quad P=(\cos\phi,\sin\phi) ,\\ &\text{and let }S\text{ be the area of }\triangle ABC . \end{align} Then \begin{align} |AB|&=|BC|=|CA|=\sqrt3,\quad \angle BAE=120^\circ,\quad \angle APB=120^\circ,\quad \angle EPA=60^\circ ,\\ \angle OAP&=\angle APO=\angle PCA=\tfrac\phi2,\quad \angle PAE=60^\circ+\tfrac\phi2,\quad \angle AEP=60^\circ-\tfrac\phi2 ,\\ S&=\tfrac{3\sqrt3}4 . \end{align}

\begin{align} D&=(-\tfrac12, \tfrac32\cot\tfrac\phi2) ,\\ |AD|^2&=|AM|^2+|DM|^2=\tfrac94\,(1+\cot^2\tfrac\phi2) =\frac9{4\sin^2\tfrac\phi2} ,\\ |AD|&=\frac3{2\sin\tfrac\phi2} ,\\ |AP|^2&=2\,(1-\cos\phi) ,\\ \triangle AEP:\quad |AE|^2&=\frac{|AP|^2\sin^2\angle EPA}{\sin^2\angle AEP} =\frac{3\,(1-\cos\phi)}{2\sin^2(60^\circ-\tfrac\phi2)} =\frac{3\,\sin^2\tfrac\phi2}{\sin^2(60^\circ-\tfrac\phi2)} ,\\ |AE|&= \frac{\sqrt3\,\sin\tfrac\phi2}{\sin(60^\circ-\tfrac\phi2)} ,\\ \triangle BCF,\triangle AFC:\quad |CF|&= \frac{2\,S}{\sqrt3\,(\sin\tfrac\phi2+\sin(60^\circ-\tfrac\phi2))} = \frac{3}{2\,(\sin\tfrac\phi2+\sin(60^\circ-\tfrac\phi2))} , \end{align}

which leads to \begin{align} E&=(1+|AE|\,\cos30^\circ,\,|AE|\,\sin30^\circ) = \left(1+ \frac{3\,\sin\tfrac\phi2}{2\sin(60^\circ-\tfrac\phi2)}, \, \frac{\sqrt3\,\sin\tfrac\phi2}{2\sin(60^\circ-\tfrac\phi2)} \right) ,\\ F&=\left( -\tfrac12+|CF|\cos(30^\circ+\tfrac\phi2) ,\, -\tfrac{\sqrt3}2+|CF|\sin(30^\circ+\tfrac\phi2) \right) \\ &= \left( -\tfrac12+ \frac{3\,\sin(60^\circ-\tfrac\phi2)}{2\,(\sin\tfrac\phi2+\sin(60^\circ-\tfrac\phi2))} ,\, -\tfrac{\sqrt3}2+ \frac{3\,\cos(60^\circ-\tfrac\phi2)}{2\,(\sin\tfrac\phi2+\sin(60^\circ-\tfrac\phi2))} \right) . \end{align}

The area of $\triangle DFE$ in terms of coordinates of $D,F,E$ is known to be expressed as

\begin{align} \tfrac12(D_xF_y+F_xE_y+E_xD_y-D_yF_x-F_yE_x-E_yD_x) . \end{align}

The following maxima code

Dx:-1/2$
Dy:3/2*cos(phi/2)/sin(phi/2)$
Fx:-1/2+3/(2*(sin(1/2*phi)+cos(1/6*%pi+1/2*phi)))*sin(%pi/3-phi/2)$
Fy:-sqrt(3)/2+3/(2*(sin(1/2*phi)+cos(1/6*%pi+1/2*phi)))*cos(%pi/3-phi/2)$
Ex:1+3*sin(phi/2)/(2*sin(%pi/3-phi/2))$
Ey:sqrt(3)*sin(phi/2)/(2*sin(%pi/3-phi/2))$
factor(trigexpand((Dx*Fy+Fx*Ey+Ex*Dy-Dy*Fx-Fy*Ex-Ey*Dx)/2));

gives this result:

\begin{align} \frac{3^{3/2}(\sin^2\tfrac\phi2-3\cos^2\tfrac\phi2)}{ 2\,(\sin\tfrac\phi2-\sqrt3\cos\tfrac\phi2) (\sin\tfrac\phi2+\sqrt3\cos\tfrac\phi2) }, \end{align}

trivially simplified further to get $\frac{3\sqrt3}2$, which is indeed, twice the area of $\triangle ABC$.

Proof for any point $P=(k,P(k)), \ \forall k \in[-\frac{\sqrt3}3s,\frac{\sqrt3}3s], \ s\in \mathbb{R}$

The area of triangles $\triangle DEF$ and $\triangle ABC$ can be written as: $\require{cancel}$ $$A_{\triangle ABC}=\pm\frac12 \det\begin{vmatrix} A_X&A_Y&1\\ B_X&B_Y&1\\ C_X&C_Y&1\\ \end{vmatrix}$$ $$ A_{\triangle DEF}=\pm\frac12 \det\begin{vmatrix} D_X&D_Y&1\\ E_X&E_Y&1\\ F_X&F_Y&1\\ \end{vmatrix}$$

For any circumcircle of an equilateral triangle, the radius is equal to $\frac{\sqrt3}3 s$, where $s$ is the side of the triangle, thus the circumcircle is defined by $$x^2+y^2=\frac13s^2$$ Which means, the $P$ is defined by $P(x)=\sqrt{\frac13s^2-x^2}$, or $\big(x,P(x)\big)$

For an equilateral triangle $\triangle ABC$ with side $s$, the vertices $A,B$ and $C$ are: $$\begin{align} A=&\Bigg(-\frac{\sqrt{-s^4+8 s^2-4}}{2 \sqrt{3} s},\frac{\sqrt{3} s^2-2 \sqrt{3}}{6 s}\Bigg)\\ B=&\Bigg(0,\frac{s}{\sqrt{3}}\Bigg)\\ C=&\Bigg(\frac{\sqrt{-s^4+8 s^2-4}}{2 \sqrt{3} s},\frac{\sqrt{3} s^2-2 \sqrt{3}}{6 s}\Bigg)\\ \end{align}$$

Then suppose we have a square matrix $M_1$, arranged as discussed above, that contains points $A,B$ and $C$, the area of the $\triangle ABC$, will be: $$A_{\triangle ABC}=\pm\frac12\det|M_1|=\frac{\left(s^2+2\right) \sqrt{-s^4+8 s^2-4}}{12 s^2}$$

However, we assume $s=1$, because it simplifies the process. Then we have: $$A_{\triangle ABC}=\frac{\sqrt{3}}{4}$$

Given $(k,P(k))$, $\forall k \in [-\frac{\sqrt3}3,\frac{\sqrt3}3]$, which defines point $P$, you get points $D,E$ and $F$ as: $$\begin{align} D=&\Bigg(\frac{-\sqrt{1-3 k^2}+3 k+1}{2 \sqrt{1-3 k^2}+6 k+4},\frac{5 \sqrt{1-3 k^2}-3 k+1}{2 \sqrt{3} \left(\sqrt{1-3 k^2}+3 k+2\right)}\Bigg)\\ E=&\Bigg(\frac{3 k}{2-2 \sqrt{1-3 k^2}},-\frac{1}{2 \sqrt{3}}\Bigg)\\ F=&\Bigg(\frac{\sqrt{1-3 k^2}+3 k-1}{2 \left(\sqrt{1-3 k^2}-3 k+2\right)},\frac{5 \sqrt{1-3 k^2}+3 k+1}{2 \sqrt{3} \left(\sqrt{1-3 k^2}-3 k+2\right)}\Bigg)\\ \end{align}$$

Suppose now we have a matrix $M_2$, arranged as discussed above, that contains the points $D,E$ and $F$, the area of $\triangle DEF$, for $s=1$, is: $$\begin{align} A_{\triangle DEF}=&\pm\frac12\det|M_2|\\ =&\frac{-2592 \sqrt{3} \sqrt{1-3 k^2} k^2+216 \sqrt{3} \sqrt{1-3 k^2}-216 \sqrt{3}}{216 \left(\sqrt{1-3 k^2}-1\right) \left(\sqrt{1-3 k^2}-3 k+2\right) \left(\sqrt{1-3 k^2}+3 k+2\right)} \end{align}$$ Simplifying that actually gives us: $$\begin{align} A_{\triangle DEF}=&\frac{-216 \sqrt{3} \left(12 \sqrt{1-3 k^2} k^2-\sqrt{1-3 k^2}+1\right)}{2\cdot-216\sqrt{3} \left(12 \sqrt{1-3 k^2} k^2-\sqrt{1-3 k^2}+1\right)}\\ =&\frac{\cancel{-216} \sqrt{3} \cancel{\left(12 \sqrt{1-3 k^2} k^2-\sqrt{1-3 k^2}+1\right)}}{2\cdot\cancel{-216} \cancel{(12 \sqrt{1-3 k^2} k^2-\sqrt{1-3 k^2}+1})}\\ =&\frac{\sqrt3}2 \end{align}$$

This goes to show that for any value of $k$, and thus, for any point $P$, the area will always be twice that of $\triangle ABC$.

Take note, however, the $k$ must be an element of $[-\frac{\sqrt3}3s,\frac{\sqrt3}3s]$