Minmax problem for polygons
Solution 1:
Addendum: Later in comments there was found a counterexample for the formula given below.
Here's an educated guess: For $n\gt3$.
$$\Phi(n)=\begin{cases}\phi(k,n-1)&n=3k\\\phi(k,n)&n=3k+1\\\phi(k+1,n)&n=3k+2\end{cases}\quad(k\in\mathbb N)$$
with
$$\phi(k,n)=\frac{8\sin^3(\pi k/n)\cos(\pi k/n)}{n\sin(2\pi/n)}\;.$$
I don't have a proof for this, but some numerical evidence and some plausibility arguments.
Similar polygons lead to the same ratio. Thus we can restrict the vertices to a compact subset of the plane. If we allow degenerate polygons (with coinciding vertices), it follows that the infimum is actually attained for some minimizing polygon.
In such a minimizing polygon, all vertices must be vertices of a maximal triangle, since we could freely move any vertex that isn't without increasing the maximal triangle area, and by moving it outward we could increase the area of the polygon and thus reduce the ratio.
If a vertex is part of only one maximal triangle, we can freely move it parallel to the opposite edge of that triangle without changing the altitude and thus the area of that triangle. This can't move the vertex outward, since again we could otherwise increase the area of the polygon without increasing the area of a maximal triangle. Thus the line connecting the two neighbouring vertices must be parallel to the opposite edge of the maximal triangle.
Now if a second one of the vertices of that maximal triangle is also only part of that one maximal triangle, the same arguments apply to that vertex. But since we've changed the direction of the edge opposite it (by moving the first point), that edge is now no longer parallel to the line connecting the vertices neighbouring the second point, and the second point can therefore now be moved outward parallel to the opposite edge without increasing the area of a maximal triangle. It follows that in any maximal triangle of a minimizing polygon, at most one of the vertices can be part of only that one maximal triangle.
On the other hand, if a vertex is part of two maximal triangles with opposite edges in different directions, it isn't free to move without increasing the area of at least one of the maximal triangles. Whether this would decrease the area ratio depends on the details of the opposing edges and the line connecting the neighbouring vertices.
Now consider regular $n$-gons with $n=3k+m$ vertices. For $m=1$, each vertex is part of three different maximal triangles, each with vertices $k$, $k$ and $k+1$ vertices apart. For $m=2$, each vertex is also part of three different maximal triangles, each with vertices $k$, $k+1$ and $k+1$ vertices apart. We can check for $n=4$ and $n=5$ that moving a vertex outward would increase the area ratio, and it's clear that it will then also do so for greater $n$. Thus in these cases a regular $n$-gon is a locally minimizing polygon. I can't exclude the possibility that there's a lower global minimum with less symmetry, but it seems unlikely, especially considering the many constraints just derived.
However, for $m=0$, each vertex is part of only one maximal triangle, so in this case the regular $n$-gon isn't a minimizing polygon. To guess what might instead be a minimizing $3k$-gon, let's consider the area ratios of regular $n$-gons. In terms of the function $\phi$ defined above, these are
$$\Phi_{\text{reg}}(n)=\begin{cases}\phi(k,n)&n=3k\\\phi(k,n)&n=3k+1\\\phi(k+1,n)&n=3k+2\end{cases}\quad(k\in\mathbb N)\;.$$
Plotting this sequence shows that $\Phi_{\text{reg}}(3k-1)\lt\Phi_{\text{reg}}(3k)\gt\Phi_{\text{reg}}(3k+1)\gt\Phi_{\text{reg}}(3k+2)$, that is, the sequence decreases except that it increases from $3k-1$ to $3k$. Thus, a plausible candidate for a minimizing $3k$-gon is a degenerate one formed by a regular $(3k-1)$-gon with one vertex repeated. This is the weakest part of the argument; though numerically I haven't found any hexagons with lower ratios than this, I can't say that the random search has been efficient enough to exclude them.
Here are algebraic expressions for some values according to the guess:
$$ \begin{array}{c|c} n&\Phi(n)\ \hline 4&\frac12\ 5&\frac1{\sqrt5}\ 6&\frac1{\sqrt5}\ 8&(2+\sqrt2)/8\ 9&(2+\sqrt2)/8\ 10&(2+\sqrt5)/10
\end{array} $$