How fat is a triangle?

I do not think using squares is a good idea. If you use instead the ratio $r/R$ of the inradius to the circumradius, you get a very clean formula:
$$ 4\sin(A/2)\sin(B/2)\sin(C/2), $$ where $A, B, C$ are the angles of the triangle. See for instance here.


The containing square

If we imagine rotating the triangle by an angle $\theta$ and measuring the width (minimum $x$ coordinate to maximum $x$ coordinate, as could be measured with a vernier caliper), we get a width function $w(\theta)$ which can be expressed in terms of the side lengths $s_1,s_2,s_3$ and side headings $\phi_1, \phi_2, \phi_3$ as $$w(\theta)=\max\left( s_1\left|\cos(\phi_1-\theta)\right|, s_2\left|\cos(\phi_2-\theta)\right|, s_3\left|\cos(\phi_3-\theta)\right| \right).$$ (Note that the maximum of those three values, representing the side that extends from the minimum $x$ coordinate to the maximum $x$ coordinate, always equals the sum of the other two.)

A square whose sides have headings $\theta$ and $\theta+\pi/2$ can fit around the triangle iff its sides have length at least $q(\theta)=\max\left(w(\theta),w(\theta+\pi/2)\right)$.  The smallest containing square is $\min_\theta q(\theta)$.

Since $q$ is the maximum of six rectified sinusoids, its minimum must be at one of the values of $\theta$ where (at least) two maximal sinusoids meet.  There are at most six such values in $[0,\pi/2]$ ($q$ is periodic with period $\pi/2$) so they can be enumerated to find the minimum.

Geometric configurations of the minimal square: For triangles having at most one angle below $\pi/4$, there are two possible configurations: (1) One triangle vertex is in a corner of the square, with the other two vertices on the two far sides, and/or (2) two vertices of the triangle are on one side of the square, with the third vertex on the opposite side.  Triangles with two angles below $\pi/4$ use a third configuration: (3) The square has the triangle's long side as its diagonal.  For example, a triangle with $(s_1,s_2,s_3)=(6,9,9)$ yields configuration (1), while $(8,9,9)$ yields configuration (2), and $(14,9,9)$ yields configuration (3).

The contained square

Since every triangle is convex, the square is inside the triangle iff its four corners are in the triangle.

If a side of the triangle is untouched by the square's corners, a homothety (centered at the vertex where the two other sides meet) can enlarge the square while keeping it in the triangle.  Therefore, the maximal square's vertices touch all three sides of the triangle.

If two of the triangle sides are touched only at their common vertex (by square corner $A$), then the third triangle side can only be touched by corner $C$ (the corner diagonally opposite $A$).  In this case, the square is not maximal, as it can be rotated and enlarged while staying inside the triangle.  Therefore, at least three of the square's corners lie on the triangle.

If two corners of the square are on the same side (say $b$) of the triangle, then the square might be maximal.  It is easy to find the largest square that sits on side $b$: Either $b$ is one of the short sides of an obtuse triangle, in which case the largest square has a vertex at the obtuse angle, or else the altitude $h_b$ to side $b$ lies inside the triangle, in which case the largest square has its other two vertices on the other two sides of the triangle, and has a side length of half the geometric mean of the lengths of $h_b$ and $b$, namely $s=\frac{h_b b}{h_b + b}$.  By finding the largest square for each of the three sides of the triangle, we can easily find the largest of these three squares, which as we will soon see is indeed the maximal possible.

The only remaining case is that three corners of the square lie in the interiors of the three sides of the triangle, with the fourth corner strictly inside the triangle.  In this case, the square can again be rotated and enlarged.  This can be seen by considering the paths of the other corners of the square as two corners slide back and forth on two sides of the triangle: The paths are elliptical.  This means that the square can be moved so that two corners stay on the triangle, and (in at least one direction) the third corner will move into the interior of the triangle, so (like the first case above) the square is not maximal.


After all those appetizer comments of mine, the OP himself is asking for : what formula describes the slimness of a triangle, based on Steiner Ellipses? So it's time to feed some meat to the public :-)
Needed for the Steiner ellipse, or best fit ellipse, or ellipse of inertia (they are all similar) are the variances (in statistics terms) of moments of inertia (in physics terms). To calculate the latter, there are three possibilities:

  1. Simplest case: all triangle weight is in the vertices $(x_1,y_1),(x_2,y_2),(x_3,y_3)$
  2. The triangle consists of three rods and all weight is in there
  3. The triangle is cut from a plate and the weight is distibuted uniformly over its area
The area point of view has been worked out in the reference Steiner Ellipses and Variances. At the last page of this reference is also the important statement that the best fit ellipse of a triangle is an inertial ellipse and is the inner Steiner ellipse. For simplicity, we shall adopt the weight at vertices case here. See the reference, or work out the second case yourself, if you want otherwise. $$ \mu_x = (x_1+x_2+x_3)/3 \quad ; \quad \mu_y = (y_1+y_2+y_3)/3 \\ \sigma_{xx} = (x_1^2+x_2^2+x_3^2)/3 - \mu_x^2 = \frac{2}{9}(x_1^2+x_2^2+x_3^2-x_1x_2-x_1x_3-x_2x_3) \\ \sigma_{yy} = (y_1^2+y_2^2+y_3^2)/3 - \mu_y^2 = \frac{2}{9}(y_1^2+y_2^2+y_3^2-y_1y_2-y_1y_3-y_2y_3) \\ \sigma_{xy} = \left[(x_1-\mu_x)(y_1-\mu_y)+(x_2-\mu_x)(y_2-\mu_y)+(x_3-\mu_x)(y_3-\mu_y)\right]/3 = \\ \frac{1}{9}(2x_1 y_1 + 2x_2 y_2 + 2x_3 y_3 - x_1 y_2 - x_2 y_1 - x_2 y_3 - x_3 y_2 - x_1 y_3 - x_3 y_1) $$ The best fit ellipse of inertia is: $$ \frac{\sigma_{yy}(x-\mu_x)^2 - 2\sigma_{xy}(x-\mu_x)(y-\mu_y) + \sigma_{xx}(y-\mu_y)^2} {\sigma_{xx}\sigma_{yy}-\sigma_{xy}^2} = 2 $$ The eigenvalues of the covariance matrix (of inertia) are related to the axes of the ellipse: $$ \left| \begin{matrix} \sigma_{xx}-\lambda & \sigma_{xy} \\ \sigma_{xy} & \sigma_{yy}-\lambda \end{matrix} \right| = 0 \qquad \Longleftrightarrow \qquad \lambda^2 - (\sigma_{xx}+\sigma_{yy})\lambda + (\sigma_{xx}\sigma_{yy}-\sigma_{xy}^2) = 0 $$ Rename (Trace and Determinant): $$ \mbox{Tr} = \sigma_{xx}+\sigma_{yy} \qquad ; \qquad \mbox{Det} = \sigma_{xx}\sigma_{yy}-\sigma_{xy}^2 $$ Then the solutions are (very much real and positive): $$ \lambda_{\pm} = \mbox{Tr}/2 \pm \sqrt{\left(\mbox{Tr}/2\right)^2 - \mbox{Det}} $$ The excentricity of the ellipse is: $$ \epsilon = \sqrt{\frac{\lambda_{+}}{\lambda_{-}}} = \frac{\sqrt{\lambda_{+}\lambda_{-}}}{\lambda_{-}} = \frac{\sqrt{\mbox{Det}}}{\lambda_{-}} = \frac{\lambda_{+}}{\sqrt{\mbox{Det}}} $$ Which is the formula you asked for (hopefully).
(Un)fortunately I have only a program that generates pictures for the case that the weight of the triangle is uniformly distributed over its area (which anyway seems to be the more interesting):

enter image description here

Update. It may be questioned if choosing one of the triangle views, as 3 vertices (1), as 3 rods (2), as a flat plate (3), has any influence on the slimness factor.
It can be easily proved that the center of gravity / midpoint of the triangle in all three cases is the same : $\mu_x = (x_1+x_2+x_3)/3 \; ; \; \mu_y = (y_1+y_2+y_3)/3$ . Now define the following quantities, where it is noted that the last two can be derived from the first one: $$ s_{xy} = (x_1-x_2)(y_1-y_2)+(x_2-x_3)(y_2-y_3)+(x_3-x_1)(y_3-y_1) \\ s_{xx} = (x_1-x_2)^2+(x_2-x_3)^2+(x_3-x_1)^2 \quad ; \quad s_{yy} = (y_1-y_2)^2+(y_2-y_3)^2+(y_3-y_1)^2 $$ Then the second order moments of inertia / variances in the vertex case can be rewritten as:
$\sigma_{xx} = s_{xx}/9$ , $\sigma_{yy} = s_{yy}/9$ , $\sigma_{xy} = x_{xy}/9$ .
Compare this with the quantities of the flat plate case in the abovementioned paper . Then it turns out that the formulas are very similar :
$\sigma_{xx} = s_{xx}/36$ , $\sigma_{yy} = s_{yy}/36$ , $\sigma_{xy} = s_{xy}/36$ .
Which means that the ellipse in the vertex case is larger than the ellipse in the flat plate case, twice as large to be precise, because $\sqrt{36/9}=2$ . In the flate plate case, we get the Steiner Inellipse and in the vertex case, we get the Steiner Outellipse . For the slimness factor, this makes no difference.

EDIT. Three Rods case. If I am right with my calculated ratio $3:6:12$ (instead of Matt's $3:8:12$) then we have the following picture for the $\color{green}{vertex}$ case, the $\color{blue}{rods}$ case and the flat $\color{red}{plate}$ case.
Conclusion: The slimness factor in all three cases is the same.

enter image description here

Coordinate independence. The formula for the Trace can be rewritten as follows: $$ \mbox{Tr} \sim (s_{xx}+s_{yy})/2 = (a^2+b^2+c^2)/2 $$ Where $a,b,c$ are the lengths of the triangle edges, as usual. The Determinant is conjectured to be proportional to the square of the area of the triangle. MAPLE is invoked to confirm this:

A := simplify(s_xxs_yy-s_xy^2);
B := simplify(((x_2-x_1)(y_3-y_1)-(x_3-x_1)*(y_2-y_1))^2);
verify(A,3*B,equal);
                           true
Hence: $$ \mbox{Det} \sim (s_{xx}s_{yy}-s_{xy}^2) = 3(2A)^2 = 12 A^2 $$ Where $A$ denotes the Area of the triangle.

Quoting a comment: I don't see, where are the angles of the triangle in this formula?
With Heron's formula for the area $A$ and some algebra we find with the above: $$ \lambda_{\pm} \sim \frac{a^2+b^2+c^2}{2} \pm \sqrt{a^4+b^4+c^4-(a^2b^2+b^2c^2+a^2c^2)} $$ The sine rule says that, with one and the same proportionality constant $K$ : $$ a = K\sin(\alpha) \quad ; \quad b = K\sin(\beta) \quad ; \quad c = K\sin(\gamma) $$ Since in the (square root of) the quotient $\;\lambda_{+}/\lambda_{-}\;$ any proportionality constant will disappear, it is indeed possible to find a more "closed" formula for the Steiner based slimness factor, as a function of the angles only : simply replace $\;a,b,c\;$ by the corresponding $\;\sin(\alpha),\sin(\beta),\sin(\gamma)$ .