Finding the distance between two triangles, one inside the other

Draw lines connecting the "respective vertices". Add up the areas to solve for $x$.

enter image description here

Area of a right triangle with the non-hypotenuse sides being $a$ and $b$ is $\frac{1}{2}(a \times b)$ while the area of a trapezium with parallel sides being $a$ and $b$ and the height being $h$ is $\frac{1}{2}h(a+b)$

$$\frac{1}{2}x(5+10) + \frac{1}{2}x(3+6) + \frac{1}{2}x(4+8) + \frac{1}{2}(3 \times 4) = \frac{1}{2} (6 \times 8)$$

$$36x + 12 = 48 \Rightarrow x = 1$$

EDIT:

Let us try to look at a slightly general case. Take a triangle and scale it to another similar triangle with the scale factor being $t$.

enter image description here

Let the sides of the inner triangle be $a$, $b$ and $c$.

The perimeter and area of the inner triangle is $P$ and $A$ respectively.

The sides of the outer triangle are $ta$, $tb$ and $tc$ while the perimeter and area are $tP$ and $t^2A$.

As before join the "respective" vertices and summing the areas give us,

$$A + \frac{1}{2}x(ta + a) + \frac{1}{2}x(tb + b) + \frac{1}{2}x(tc + c) = t^2A$$

$$A + \frac{1}{2}x(t+1)P = t^2A \Rightarrow \frac{1}{2}x(t+1)P = (t^2-1)A$$

$$x = \frac{2A}{P}(t-1)$$

In the problem asked, $t=2$ with $P = 2A = 12$ and hence we get $x=1$.

Also, $t=1$ gives $x=0$ as expected.

This also gives a nice proof that the radius of the incircle of a triangle is $$r_{in} = \frac{2A}{P}$$

This is got by plugging in $t=0$ and realizing that $\left| x \right|$ is nothing but the radius of the incircle.

Hence, $$x = (t-1)r_{in}$$


Algebraic Solution:

Extend the lines making up the side length $3$ of the inner triangle until it hits the walls of the larger one, and consider the triangle formed. What is the length of this line particular line? Well it is $$x+3+(?)$$ where $(?)=\frac{5}{4}x$ by similar triangles. What about the base line? It has length $8-x$. And the hypotenuse? It has length $10-x\frac{5}{4}$ by similar triangles again. Then by the Pythagorean Theorem, $$\left(10-x\frac{5}{4}\right)^{2}=\left(8-x\right)^{2}+\left(3+\frac{9}{4}x\right)^{2}$$ and solving this quadratic yields $x=1$ as a solution.

Quick Solution:

Notice the side lengths of the larger triangle sum to equal its area, 24. (That is why we get 1) Draw straight tubes, a $x\times 6$ tube, a $x\times 8$ tube and a $x\times 10$ tube along the sides on the triangle, on the inside so that all of the area of the triangle except the missing middle piece is covered. (The middle piece is the smaller triangle) Then the overlapping parts, and the parts hanging outside the triangle can be moved around to make up the smaller inside triangle which is missing. Hence the areas are equal so $8x+6x+10x=\frac{6\cdot 8}{2}=24$ so $x=1$.

Remark: I also hope this is understandable without a picture.