An ancient Japanese geometry problem: Three circles of equal radius inscribed in an isosceles triangle.

Let's define some coordinates.

\begin{align*} A &= (-1,0) & P &= C+a(B-C) & C_1 &= (c,r) \\ C &= (1,0) & H &= A+b(P-A) & C_2 &= (d,e) \\ B &= (0,y) &&& C_3 &= (0,f) \end{align*}

Here $C_{1,2,3}$ are the centers of the three circles, from bottom to top. The above coordinates contain $9$ variables: $a\dots f,r,x,y$. You can express the tangentialities based on these:

\begin{align*} AP,C_1: 0 &= -ac^2y+ar^2y-2acr-2acy-2ar+4cr-ay+4r \\ CH,C_1: 0 &= -abc^2y+abr^2y-2abcr+2abcy+2abr+4bcr-aby-4br-4cr+4r \\ AP,C_2: 0 &= -a^2d^2y^2+a^2r^2y^2-2a^2dey-2a^2dy^2-a^2e^2+a^2r^2-2a^2ey\\&\phantom=+4adey-a^2y^2+4ae^2-4ar^2+4aey-4e^2+4r^2 \\ CH,C_2: 0 &= -a^2b^2d^2y^2+a^2b^2r^2y^2-2a^2b^2dey+2a^2b^2dy^2-a^2b^2e^2+a^2b^2r^2\\&\phantom=+2a^2b^2ey+4ab^2dey-a^2b^2y^2+4ab^2e^2-4ab^2r^2-4ab^2ey-4abdey\\&\phantom=-4abe^2-4b^2e^2+4abr^2+4b^2r^2+4abey+8be^2-8br^2-4e^2+4r^2 \\ CB,C_2: 0 &= -d^2y^2+r^2y^2-2dey+2dy^2-e^2+r^2+2ey-y^2\\ AB,C_3: 0 &= r^2y^2-f^2+r^2+2fy-y^2\\ AP,C_3: 0 &= a^2r^2y^2-a^2f^2+a^2r^2-2a^2fy-a^2y^2+4af^2-4ar^2+4afy-4f^2+4r^2 \end{align*}

There is one more equation, which relates $X$ to these variables:

$$X^2 = a^2b^2y^2+a^2b^2-4ab^2+4ab+4b^2-8b+4$$

I obtained these equations in the following way: I homogenized the two points defining a line, and computed their cross product to represent the line joining them. I also expressed the circles in a projective way, as symmetric $3\times 3$ matrices representing the quadratic form of a conic section in homogeneous coordinates. Taking the adjunct of that matrix I obtained a description of the dual conic, in terms of tangent lines instead of incident points. So I multiplied that matrix with the vector of the line from both sides, let my CAS handle all the gory details, and pasted the result above. The final equation is simply the expanded form of $X^2=\langle C-H,C-H\rangle$.

It's also good to have one approximate solution at hand. I constructed one using Cinderella, using a small script to adjust one parameter using bisection. I obtained a configuration with

\begin{align*} a&\approx\phantom+0.60827488013287\\ b&\approx\phantom+0.5392413189553\\ c&\approx-0.199319828889346\\ d&\approx\phantom+0.284316034733135\\ e&\approx\phantom+0.803339370375978\\ f&\approx\phantom+1.443050448525532\\ r&\approx\phantom+0.327733253018358\\ X&\approx\phantom+1.451198842282192\\ y&\approx\phantom+2.25 \end{align*}

Now one can try to eliminate variables $a$ through $f$ and $y$ using standard elimination techniques. I used resultants, and furthermore, after each resultant computation I factorized the result, and kept only the single factor which almost vanished for my approximate example data. In the end, I obtained the solution

$$r^6 + 3r^4X^2 + r^2X^4 + 4r^4X - 2r^2X^3 + 5r^4 - 8r^2X^2 - X^4 - 4r^2X + 4X^3 + 4r^2 - 4X = 0$$

Obviously it would have ben impossible to follow this approach without massive support from a CAS, Sage in my case.

The result is not homogeneous, though, which means that it makes implicit reference to the fact that I fixed the length of edge $AC$ as $2$. The way I see it, the problem can't be answered without knowing either the length of one of the edges, or the ratio $AB/AC$. Is there some information missing in the question?

Fixing one length appears slightly inelegant. If we have to introduce an extra parameter, it would be better to choose a dimensionless ratio for this. The ratio $AB/AC$ mentioned above would lead to a somewhat longer expression, so I prefer to use $s$ to denote the ratio between height and base length of the triangle. $s$ stands for “shape” since that ratio only depends on the shape of the triangle, not its size. In the above coordinates, $s$ would be $y/2$. Using this parameter, and following the same approach outlined above, you'd end up with the condition

$$ 16s^5r^{10} - (16s^5 + 28s^3)r^8X^2 - (8s^4 + 4s^2)r^7X^3 + (4s^5 + s^3 + 12s)r^6X^4 + (12s^4 + 11s^2 + 4)r^5X^5 + (25s^3 + 15s)r^4X^6 - (4s^4 - 10s^2 - 5)r^3X^7 - (11s^3 + s)r^2X^8 - 3s^2rX^9 + s^3X^{10} = 0 $$

This condition is homogeneous in $r,X$ since their powers always add up to $10$, as is to be expected since now everything is invariant under scaling.

With right angle

It appears that the original statement of this problem had $AP\perp CH$. In my world, this translates to $\langle A-P, C-H\rangle=0$ or

$$a^2by^2 + a^2b - 4ab + 2a + 4b - 4=0$$

Adding this to my system of equations, one can solve for all variables. It turns out that all the variables are even rational numbers:

\begin{align*} a &= \frac{14}{25} & b &= \frac{1}{2} & c &= -\frac{1}{5} \\[2ex] d &= \frac{7}{25} & e &= \frac{26}{25} & f &= 2 \\[2ex] r &= \frac{2}{5} & X &= \frac{8}{5} & y &= \frac{24}{7} \end{align*}

You can verify that this satisfies all the equations stated above. So if you choose your coordinate system in such a way that the points $A$ and $C$ are at the given coordinates, then the other points will have the coordinates I just stated. It is trivial to read $X = 4r$ from this. You may notice that for $s=12/7$ the equation given above factors, and $X-4r$ is one of its factors.

If you scale everything by $175$ you even get integer coordinates:

\begin{align*} A &= (-175,0) & P &= (77,336) & C_1 &= (-35,70) \\ C &= (175,0) & H &= (-49,168) & C_2 &= (49,182) \\ B &= (0,600) &&& C_3 &= (0,350) \end{align*}

In this diagram, $$|\overline{AB}| = |\overline{CB}| = a \qquad |\overline{AC}| = 2b \qquad |\overline{AD}| = d \qquad |\overline{CE}| = e$$ Also, $$x = \frac{|\overline{BD}|}{|\overline{BC}|} \qquad y = \frac{|\overline{AE}|}{|\overline{AD}|} \qquad \angle AEC = 2\theta$$ (where $\angle AEC$ may or may not be a right angle). Writing $T$ for the area of $\triangle ABC$, we see that $$|\triangle ABD| = Tx \qquad |\triangle CDE| = T(1-x)(1-y) \qquad |\triangle ACE| = T(1-x)y$$

The relation $$\text{inradius}\cdot\text{perimeter} = 2\cdot\text{area}$$ implies $$\begin{align} r\;\left(\; a+ax+d \;\right) &= 2\;|\triangle ABC| = 2 T x \tag{1}\\ r\;\left(\; a (1-x) + d (1-y) + e \;\right) &= 2\;|\triangle CDE| = 2T(1-x)(1-y) \tag{2}\\ r\;\left(\; e + 2 b + d y \;\right) &= 2\;|\triangle ACE| = 2T (1-x)y \tag{3} \end{align}$$

We also have $$T^2 = b^2 ( a^2 - b^2 ) \tag{4}$$

Stewart's Theorem applied to Cevians $d$ and $e$ of $\triangle ABC$ and $\triangle ACD$, ultimately gives us that $$\begin{align} d^2 &= a^2 ( 1 - x )^2\phantom{y} + 4 b^2 x \tag{5}\\ e^2 + d^2 y ( 1 - y ) &= a^2 ( 1 - x )^2 y + 4 b^2 ( 1 - y ) \tag{6} \end{align}$$ Finally, the Law of Cosines provides this relation $$4 b^2 = d^2 y^2 + e^2 - 2 d e y \cos2\theta \tag{7}$$

Thus, we have seven equations in seven parameters: $a$, $b$, $d$, $e$, $x$, $y$, $T$. There is hope of eliminating six parameters to achieve a relation $F(r,e,\theta) = 0$.

The elimination process is tedious, and I haven't found a particularly "good" path through it. However, I'll note that we can solve $(1)$, $(2)$, $(3)$ that are linear in $a$, $b$, $T$ to get:

$$a = \frac{ex - d(1-y)(1-2x)}{(1 - x)(1 - y - x y )} \qquad b = \frac{2 e (1+x)y - e - d (1-2x)y}{2(1 - y - x y)} \tag{$\star$}$$ $$T = r\;\frac{e (1+x) + 2 d x - d (1+x) y}{2( 1 - x)( 1 - y - x y)}$$

This immediately reduces our burden a bit. Also, we can solve $(5)$ and $(6)$ for $a^2$ and $b^2$ $$a^2 =\frac{d^2 (1-y-xy+xy^2) - e^2 x}{(1 - x)^2(1 - y - x y)} \qquad b^2 = \frac{e^2 - d^2 y^2}{4(1 - y - x y)} \tag{$\star\star$}$$ which helps in quickly establishing further reductions. (For instance, one can equation $a^2$ from $(\star)$ with $a^2$ from $(\star\star)$.)

With the help of Mathematica, I was able to wade through a river of resultants to get to this relation

$$\begin{align} \phantom{\cdot} &\left(\;2 \hat{r}^3 + \hat{r}^2 \hat{e}\;(1-3\sin^2\theta) - 2 \hat{r} \hat{e}^2 \sin^2\theta + \hat{e}^3 \sin^4\theta\;\right) \\ \cdot &\left(\;2 \hat{r}^3 - \hat{r}^2 \hat{e}\;(1-3\sin^2\theta)- 2 \hat{r} \hat{e}^2 \sin^2\theta - \hat{e}^3 \sin^4\theta\;\right) = 0 \end{align}$$

where $\hat{r} := r\sin\theta$ and $\hat{e} := e\cos\theta$.

(Of course, since the factors are merely cubics, we can solve for roots $\hat{r}$ explicitly. This is left as an exercise for the reader.)

When $\angle AEC$ is a right angle, $\theta = \pi/4$ and the equation reduces to

$$(4r - e) (2r^2 - e^2)\cdot(4r + e) (2r^2 - e^2) = 0$$

Since $r$ and $e$ are non-negative (and $r \leq e/2$), we conclude $r = e/4$.