Proof that every polygon with an inscribed circle is convex?

Thanks to everyone who suggested approaches to this problem. In the end, none of the suggested approaches fit into the axiomatic framework that I was working in, so I had to write up my own rather laborious proof. It's a bit long to post here, but in order to close this question, I just want to post the reference and a quick summary of the approach.

You can find the complete proof in my textbook Axiomatic Geometry (Theorem 14.31). The basic idea is first to prove the following lemma:

Lemma. Let $\mathscr P$ be a polygon circumscribed about a circle $\mathscr C$. Suppose $A$ is any vertex of $\mathscr P$, and $E$ and $F$ are the points of tangency of the two edges containing $A$. Then there are no points of $\mathscr P$ in the interior of $\triangle AEF$.

To prove that a tangential polygon $\mathscr P$ must be convex, the basic idea is to show that if $\ell$ is any edge line of $\mathscr P$, then there can't be any vertices of $\mathscr P$ on the "wrong" side of $\ell$ (the side not containing the inscribed circle), because that would violate the lemma.


The definition of simple establishes that the polygon itself equals the intersection of the half-planes tangent to the circle at the points where the polygon's sides contact the circle. Any nonempty intersection of halfplanes is convex.


edit 2: Suppose that a simple polygon has an inscribed circle. Without loss of generality, pick a "first" edge and let the circle be on the "right" side of that edge. The angle that is on the same side as the circle--that is, to the right--between the first edge and the second edge must have measure less than 180°. Similarly, the angle between the second and third edges that is also on the right must also have measure less than 180°, and so on, so that all of the angles on the right side of the polygon's perimeter (whether it is the inside or the outside) must have measure less than 180° and all of the angles on the left side must have measure greater than 180°.

Since the sum of the interior angles of a simple polygon with n sides is 180°(n – 2), the average measure of an interior angle of a simple polygon with n sides is 180° – 360°/n, which is strictly less than 180°. So, since the angles on the left side of the perimeter all have measure greater than 180°, their average is greater than 180°, so the left cannot be the interior of the polygon and the right side must be the interior, so the inscribed circle must be in the interior of the polygon and the internal angles all have measure less than 180°.


original answer:

Assuming the polygon is non-self-intersecting, then two consecutive sides of the polygon correspond to two consecutive points on tangency on the circle, with the angle formed by the two sides subtending the minor arc between the points of tangency. The measure of the angle of the polygon is half the difference between the measure of the major and minor arcs between the points of tangency. The greatest possible difference would be the degenerate case where the minor arc has measure 0° and the major arc has measure 360°, giving the angle measure 180°; for non-degenerate cases, the difference in the arc measures must be less than 360°, so the angle measure must be less than 180°. This applies to every pair of consecutive sides, so every interior angle of the polygon has measure less than 180°, so the polygon is convex.


edit: Alternately, and blatantly assuming that the inscribed circle must be in the interior of the polygon, suppose that a polygon is not convex, so that there is a vertex at which the interior angle has measure greater than 180°. For a circle to be tangent to the two sides that meet at that vertex, the circle must be exterior to the polygon at that vertex (so as to be in the non-reflex side of that angle) and thus cannot be the inscribed circle.