Two circles inside a right angled triangle!

Writing $a := |BC|$, $b := |CA|$, $c := |AB| = \sqrt{a^2+b^2}$, and $r = |PE| = |PF|$ (so that $|PD| = 3r$), we have $$\begin{align} |\triangle ABC| &= |\triangle ABP| + |\triangle BCP| + |\triangle CAP| \\[4pt] \implies \qquad \frac{1}{2} |BC||CA| &= \frac{1}{2} \left(\; |AB| |PF| + |BC||PD| + |CA||PE| \;\right) \\[4pt] \implies \qquad a b &= c r + 3 a r + b r = r ( 3 a + b + c )\\[6pt] \implies \qquad r &= \frac{ab}{3 a + b + c} = \frac{ab}{3 a + b + \sqrt{a^2+b^2}} \end{align}$$

To address @DanielV's suggestion of generalizing to higher dimensions, consider a right-corner tetrahedron $OABC$, with right corner at $O$ and edge lengths $a := |OA|$, $b := |OB|$, $c := |OC|$. (Note that I'm changing notation slightly from the above.) Let a sphere with center $P$ and radius $r$ be tangent to the faces around vertex $A$, and let a congruent sphere (tangent to the first) be tangent to the faces around vertex $O$. Then $P$ has distance $r$ from faces $\triangle OAB$, $\triangle OCA$, $\triangle ABC$ (the ones touching $A$), and distance $3r$ from face $\triangle OBC$ (the one opposite $A$).

Here's a poor attempt at a diagram:

(In this case, the altitudes from $P$ are color-coded to match their parallel counterparts through $O$. The black altitude is to face $\triangle ABC$.)

Thus,

$$\begin{align} |OABC| &= |OABP| + |OBCP| + |OCAP| + |ABCP| \\[4pt] \implies \qquad \frac{1}{6}a b c &= \frac{1}{3}\left(\; r\;|\triangle OAB| + r \;|\triangle OCA| + r\;|\triangle ABC| + 3r\;|\triangle OBC| \;\right) \\[4pt] &= \frac{1}{3}r \cdot \frac{1}{2} \left(\; a b + c a + 3 b c + 2\;|\triangle ABC| \;\right) \\[6pt] \implies \qquad r &= \frac{abc}{3bc + ab + ca + 2\;|\triangle ABC|} \qquad (\star) \end{align}$$

Fun fact: The Pythagorean Theorem for Right-Corner Tetrahedra says that $$|\triangle ABC|^2 = |\triangle OBC|^2 + |\triangle OCA|^2 + |\triangle OAB|^2$$ so that we have $$|\triangle ABC| = \frac{1}{2} \sqrt{\; b^2 c^2 + c^2 a^2 + a^2 b^2 \;}$$ and $(\star)$ becomes $$r = \frac{abc}{3bc + ab + ca + \sqrt{\; b^2 c^2 + c^2 a^2 + a^2 b^2 \;}}$$

In $4$-dimensional space (where there's an analogous Pythagorean Theorem, as there is in any-dimensional space), we have $$r = \frac{abcd}{3bcd + acd + abd + abc + \sqrt{\;b^2 c^2 d^2 + a^2 c^2 d^2 + a^2 b^2 d^2 + a^2 b^2 c^2\;}}$$ and so forth.

Incidentally, the matching-notation version of the initial answer is $$r = \frac{ab}{3b + a + \sqrt{\;b^2 + a^2\;}}$$

Assuming the corner of the triangle is the origin of a Cartesian plane, the line of the hypotenuse is $$y = -\frac{a}{b} x + a$$

The center of the second circle is at $\begin{bmatrix} 3r \\ r \end{bmatrix}$.

The radius of the second circle is the directed vector $\begin{bmatrix} ra \\ rb \end{bmatrix}\frac{1}{\sqrt{a^2 + b^2}}$.

Altogether :

$$\frac{rb}{\sqrt{a^2 + b^2}} + r = -\frac{a}{b}\left(3r + \frac{ra}{\sqrt{a^2 + b^2}}\right) + a$$

$$r = \frac{ab}{ 3a + b + \sqrt{a^2 + b^2}}$$

Since Blue showed an elegant volume based approach, I guess I'll try a multidimensional coordinate based approach. Suppose the origin $\begin{bmatrix} 0 \\ 0 \\ 0 \\ \vdots \end{bmatrix}$ and the points $\begin{bmatrix} \mathcal{l}_0\\ 0 \\ 0 \\ \vdots \end{bmatrix}$, $\begin{bmatrix} 0 \\ \mathcal{l}_1 \\ 0 \\ \vdots \end{bmatrix}$, $\begin{bmatrix} 0 \\ 0 \\ \mathcal{l}_2 \\\vdots \end{bmatrix}$ etc form a multidimensional right triangle. Suppose one hypersphere is tucked in at the origin and one hypersphere is tucked in at the corner along the first axis.

The hypotenuse plane of the triangle is $$\frac{x_0}{\mathcal{l}_0} + \frac{x_1}{\mathcal{l}_1} + \frac{x_2}{\mathcal{l}_2} + \dots = 1 \tag{Plane equation}$$

Or equivalently, using $\circ$ for dot product and $N = \begin{bmatrix} \frac{1}{\mathcal{l}_0} \\ \frac{1}{\mathcal{l}_1} \\ \frac{1}{\mathcal{l}_2} \\ \vdots \end{bmatrix}$, then the plane is: $$N \circ X = 1 \tag{Plane equation with dot product}$$

The center of the second hypershpere is $$c = \begin{bmatrix} 3r \\ r \\ r \\ \vdots\end{bmatrix} = rc_0 \tag{Center of second hypersphere}$$

The vector directed from the center of the second hypersphere to the hypotenuse plane is $$r_2 = r\frac{N}{|N|}\tag{Directed radius to hypotenuse}$$

Altogether: $$N \circ (c + r_2) = 1$$ $$N \circ \left(rc_0 + r\frac N {|N|}\right) = 1$$ $$\begin{align} r &= \frac{1}{N \circ c_0 + |N|} \\ &= \frac{1}{3l_0^{-1} + l_1^{-1} + l_2^{-1} \dots + \sqrt{l_0^{-2} + l_1^{-2} + l_2^{-2} \dots}} \end{align}$$

Note this is the same answer as Blue, but with $\mathcal{l}_0\mathcal{l}_1 \mathcal{l}_2 \dots$ factored out of the numerator and denominator.

Alternative solution:

Recall that for any triangle $\triangle ABC$, the area of the triangle equals the product of the inradius and its semiperimeter; i.e., $|\triangle ABC| = rs$, where $s = (a+b+c)/2$. Therefore, given legs $a, b$, $c = \sqrt{a^2+b^2}$, and $$r = \frac{2|\triangle ABC|}{a+b+c} = \frac{ab}{a+b+\sqrt{a^2+b^2}}.$$ Draw the tangent line to the two circles at their common point of tangency: this creates a smaller similar triangle with scaling factor $\frac{b-2\rho}{b}$ where $\rho$ is the common radius of the two circles. If $r$ is the inradius of $|\triangle ABC|$ as shown, then $$\frac{\rho}{r} = \frac{b - 2\rho}{b}.$$ Putting all of this together, we find $$\rho = \frac{br}{b+2r} = \frac{a b}{3a+b+c} = \frac{ab}{3a+b+\sqrt{a^2+b^2}}.$$ This method easily generalizes to more than two congruent circles tangent to one side: if $n$ circles are arranged along the leg of length $b$, then it is straightforward to find that $\rho = \frac{ab}{(2n-1)a + b + \sqrt{a^2+b^2}}$.

Other people have answered your question, but I'll provide the "answer" for the other direction:

What about determining $a,b$ from $r$?

You can't - $a,b$ are not uniquely determined by $r$. Draw the two circles first, then draw the legs a and b as infinite lines. And line tangent to the right most circle will intersect lines A and B creating a right triangle, but for different tangents $a$ and $b$ will be different and r hasn't changed.

You can't even turn this into an interesting question by asking for all the possible ways to do it, because it turns out that these circles don't actually impose any restriction - since any angels can be achieved by the tangent line, any right triangle can. All $r$ does is create a scaling factor. $a$ can take on any value in $(2r,\infty)$ and $b$ the corresponding value in $(4r,\infty)$