Smallest square containing a given triangle
Solution 1:
I was thinking along similar lines to those of TonyK, but still different. Let some triangle be given. Let $a$ denote the length of the longest side and $b$ the length of the second longest. Let the angle between those to sides be denoted $\theta$. For $b$ to be longer than the third side we must have $b\cdot\cos\theta\geq a/2$. Then we represent the triangle by the vectors (see Dynamic GeoGebra Applet) $$ \vec u=a\cdot\vec e_{t}\qquad\text{and}\qquad\vec v=b\cdot\vec e_{t+\theta} $$ where $\vec e_s=\begin{pmatrix}\cos s \\ \sin s\end{pmatrix}$ has been defined as the unit vector pointing in the direction of the angle $s$.
Now we start rotating the triangle by changing the value of $t$. For each value of the rotation angle $t$ we form the minimal square with sides parallel to the axes that contains the triangle/the vectors. The size of the minimal squares will be periodic with period $\pi/2$. Thus we only need to consider rotations $t\in[0,\pi/2]$.
To determine the side of the minimal square for a given $t$ let us consider the four functions $$ \begin{align} \Delta x_1(t)&=a\cdot\cos t \\ \Delta x_2(t)&=a\cdot\cos(t)-b\cdot\cos(t+\theta) \\ \Delta y_1(t)&=a\cdot\sin t \\ \Delta y_2(t)&=b\cdot\sin(t+\theta) \end{align} $$ All of the above are either $x$- og $y$-differences found in the triangle for the current value of $t$. If $\square(t)$ denotes the side length of the minimal axes-parallel square containing the triangle for the current $t$ we then have $$ \square(t)=\max\{\Delta x_1(t),\Delta x_2(t),\Delta y_1(t),\Delta y_2(t)\} $$ and the optimal (minimal) value of $\square(t)$ corresponds to one of the intersection points of at least two of the four difference functions. So it is all about solving each of the equations $$ \begin{align} \Delta x_1(t)&=\Delta x_2(t)\\ \Delta x_1(t)&=\Delta y_1(t)\\ \Delta x_1(t)&=\Delta y_2(t)\\ \Delta x_2(t)&=\Delta y_1(t)\\ \Delta x_2(t)&=\Delta y_2(t)\leftarrow\text{The equation suggested by TonyK}\\ \Delta y_1(t)&=\Delta y_2(t) \end{align} $$ For each triangle this gives (at most) six $t$ values in the interval $[0,\pi/2]$ of which one will be the optimal one.
The only general statements I have found from drawing these situations and plotting the four functions so far, is that $\Delta x_1=\Delta x_2$ can be left out and at least one vertex of the triangle will be placed in a vertex of the optimal containing square.
Also I have found that there are far more complex situations than those both the OP and TonyK have come up with.
Solution 2:
Updated to add: It seems that this is wrong. String's answer tells you why.
If two angles are $\le 45^\circ$, then use your heuristic 1. Otherwise, put the smallest angle (call it $\varphi$) at the origin $O$, with side $OX$ of length $a$ along the positive $x$-axis and the other side $OY$, of length $b$, in the first quadrant.
Now rotate the triangle anticlockwise by an angle $\theta$, until the $x$-coordinate of $X$ equals the $y$-coordinate of $Y$. When this happens, all three points lie on a square, two of whose sides lie along the coordinate axes. I think (but haven't proven) that this square is minimal.
The $x$-coordinate of $X$ is $a \cos \theta$, and the $y$-coordinate of Y is $b \sin(\theta+\varphi)$. Putting these equal gives
$$a \cos \theta = b (\sin \theta \cos \varphi + \cos \theta \sin \varphi)$$
or
$$\theta = \tan^{-1} \frac{a - b\sin\varphi}{b\cos\varphi}$$
Solution 3:
No, it's not always correct. One example that fails is a tall, narrow isosceles triangle (for example, side lengths 1, 10, 10). Positioning the triangle diagonally in the square is much better than positioning it vertically.