Find the area of ​region ABC.

For reference: In figure $T$ and $K$ are points of tangency, $MT = a$ and $KN = b$; calculate area of ​​region $ABC$.

(Answer:$2\sqrt{ab}(\sqrt a+\sqrt b)^2$)

My progress:

$$S_{ABC} = p \cdot r = \frac{r \cdot (AB+BC+AC)}{2}\\ AC +2R = AB+BC\\ S_{ABC} = AG \cdot GC \qquad \text{(property)} \\ S_{ABC} = (AC+R)R \qquad \text{(property)} \\ OTBQ:~\text{square} \implies TK = R\sqrt2 \\ \ldots ?$$

I'm not able to use segments a and b in the resolution

Here is a geometrical solution without much algebra.

For a right triangle, if you draw a line through the points of tangency of the incircle with the perpendicular sides, it does bisect the arcs of the circumcircle on both sides. In other words, $M$ and $N$ are midpoints of minor arcs $AB$ and $BC$ respectively. At the end of the answer, I have shown a proof.

With that, note that $\triangle BTM \sim \triangle NKB$. That leads to,

$\frac{r}{a} = \frac{b}{r} \implies r = \sqrt{ab}$

As $FM$ is perpendicular bisector of $AB$ and $FN$ is perpendicular bisector of $BC$,

$\frac{AB}{2} = \sqrt{ab} + \frac{a}{\sqrt2}$ and $\frac{BC}{2} = \sqrt{ab} + \frac{b}{\sqrt2}$

As we know $AB$ and $BC$ in terms of $a$ and $b$, we are done, for $S_{\triangle ABC} = \frac 12 \cdot AB \cdot BC$.

Proof of the property that I used in the above answer -

Say $M$ and $N$ are midpoints of the arcs $AB$ and $BC$ and segment $MN$ intersects $AB$ and $BC$ at $T$ and $K$ respectively.

$\angle BIN = 45^\circ + \angle A/2$ So, $\angle KPN = 90^\circ + \angle A/2$
Also, $\angle PNK = \angle C/2$

That leads to $\angle BKM = \angle PKN = 45^\circ$

Also note that $\angle INK = \angle ICK = \angle C / 2$ so $ICNK$ is cyclic and therefore $\angle KIN = \angle KCN = \angle A / 2$

That leads to $\angle IKM = \angle A / 2 + \angle C / 2 = 45^\circ$

$\angle BKI = \angle BKM + \angle IKM = 90^\circ$.

So $K$ must be point of tangency of incircle with side $BC$. Finally since $KI \parallel BT$ and $BT = BK = KI$, $T$ is the point of tangency of incircle with side $AB$.

I continue your process and we already have $\text{TK}=\sqrt{2} R$.

According to intersecting chords theorem, we have $$\text{AT} \cdot\text{TB}=\text{MT}\cdot \text{TN}$$$$\text{BK} \cdot\text{KC}=\text{KN} \cdot \text{MK}$$ So $$\text{AT}=\sqrt{2} a+\frac{\text{ab}}{R}$$$$\text{KC}=\sqrt{2} b+\frac{\text{ab}}{R}$$ Because $S_{ABC}$ is a rectangular triangle, we have $\text{AB}^2+\text{BC}^2=\text{AC}^2$ and $$\text{BC}=\text{BC}+\text{KC}=\text{KC}+R$$$$ \text{AB}=\text{AT}+\text{TB}=\text{AT}+R$$$$\text{AC}=\text{AG}+\text{GC}=\text{AT}+\text{KC}$$ So solving the equation we get $$R=\sqrt{\text{ab}}$$ Therefore $$S_{ABC} = \frac{\text{AB}\cdot \text{BC}}{2}$$$$=\frac{1}{2} \left(\sqrt{2} a+2 \sqrt{\text{ab}}\right) \left(\sqrt{2} b+2 \sqrt{\text{ab}}\right)$$$$=\boxed{\sqrt{2} \sqrt{\text{ab}} (a+b)+3 a b}$$ I think the answer you give might be wrong.

Hint. Use Power of a point in combination with well-known metric relations of $\triangle ABC$ with regard to the incircle and its contacts points.

I will denote your segments $a$, $b$ with $x,y$ respectively, and will let $a,b,c$ for the side lengths of $\triangle ABC$, as usual. Also, let $s=\frac12(a+b+c)$ denote the semiperimeter of $\triangle ABC$. Using Power of a point

$$\begin{align*}AT\cdot TB=MT\cdot TN&\iff (s-a)(s-b)=x\cdot (\sqrt2\cdot(s-b)+y)\\BK\cdot KC=MK\cdot KN&\iff (s-b)(s-c)=(x+\sqrt{2}\cdot (s-b))\cdot y\end{align*}$$

I will let you fill in the missing details. Now, you have two equations, where you can solve for $x,y$ in terms of $a,b,c$ (notice: you might want to solve for $x+y, xy$ instead). Finally, what you want to prove is

$$ac=4\sqrt{xy}\cdot (\sqrt{x}+\sqrt{y})^2$$