Convexifying a Concave Polygon by Reflections
Let $P$ be a given concave polygon and consider the following procedure. Take the convex hull $Q$ of $P$ and consider a side $AB$ of $Q$ which is not a side of $P$. Then, let $P'$ be the polygon obtained by $P$ by replacing the polygonal path of $P$ from $A$ to $B$ which has no vertex in common with those of $Q$ other than $A$ and $B$ with its image in the reflection with respect to $AB$. If $P'$ is a convex polygon, then the procedure stops, otherwise replace $P$ by $P'$ and repeat the process.
Does this procedure end after finitely many steps, for every initial concave polygon $P$ and every (or at least some) possible choice of the sides with respect to which to take the reflections?
My intuition, shaped by the many concrete examples which I have carefully examined, strongly suggests that the answer is positive, but when I haved tried to sketch a proof, I got completely stuck.
$\mathbf{Remark}$. Just note that the polygons $P$ and $P'$ have the same perimeter, while $P'$ has a larger area than $P$. Actually, this procedure was described by David in his answer to the post Proving that the regular n-gon maximizes area for fixed perimeter, in which the isoperimetric problem for n-gons is discussed. Even though the eventual finiteness of the procedure is irrelevant to the solution there given, I think this issue has its own interest.
Solution 1:
You'd think this would be relatively easy to analyze and prove. The flips preserve edge length and order, and increase area. But attempts to proceed further end up in the weeds.
In case of emergency, break glass and go to the internet and find the following paper
Demaine, Gassend, O’Rourke, Toussaint, All Polygons Flip Finitely. . . Right?
Abstract. Every simple planar polygon can undergo only a finite number of pocket flips before becoming convex. Since Erdős posed this finiteness as an open problem in 1935, several independent purported proofs have been published. However, we uncover a plethora of errors, gaps, and omissions in these arguments, leaving only two proofs without flaws and no proof that is fully detailed. Fortunately, the result remains true, and we provide a new, simple (and correct) proof. In addition, our proof handles non-simple polygons with no vertices of turn angle 180◦, establishing a new result and opening several new directions.
The paper goes into extensive detail about the history of the problem and proofs, and before ascending the climb to its own proof finds the dead bodies of flawed proofs of yore on the way.
You'd think that there would be relatively plausible conjectures about the maximum number of reflections required to get to convexity (e.g. $2n$ flips for an $n$-gon), but there is no such limit, even for quadrangles (https://www.cs.mcgill.ca/~cs507/projects/1998/mas/main2.html).