Infinite Sequence of Inscribed Pentagrams - Where does it converge?
Solution 1:
Your intuition about a projective transform was correct: a projective transformation relates one iteration to the next, they don't even skip a step. A quick experiment with Cinderella verified this pretty easily. By now I even have a proof, so this answer differs greatly from previous versions.
Mapping corners for one step
This section is a proof that the transformation mapping the outer pentagram to the inner one is in fact a projective transformation. If you prefer to simply take my word for it, feel free to skip to the next section.
A projective transformation is uniquely defined by the images of four points, no three of which may be collinear. So suppose you define the projective map $T$ using the following four points:
\begin{align*} A_1&\mapsto A_2& B_1&\mapsto B_2& C_1&\mapsto C_2& D_1&\mapsto D_2 \end{align*}
Now you know that $T$ preserves incidences, so (using $\vee$ to denote “join” and $\wedge$ to denote “meet”) you get
\begin{align*} T(E_2) &= T\bigl((A_1\vee B_1)\wedge(C_1\vee D_1)\bigr) \\&=\bigl(T(A_1)\vee T(B_1)\bigr)\wedge(T(C_1)\vee T(D_1)\bigr) \\&=(A_2\vee B_2)\wedge(C_2\vee D_2) = E_3 \end{align*}
You may also know that a projective transformation will preserve cross ratios. Therefore you get $(A_1,B_1;E_2,C_2)=(A_2,B_2;E_3,T(C_2))$. So once you know $A_2$, $B_2$ and $E_3$ (as we do), the point $T(C_2)$ with relations to these has to be fixed. You can construct this point, using the fact that a perspectivity will preserve cross ratios. A perspectivity centered at $D_2$ will map
\begin{align*} A_1&\mapsto B_2& E_2&\mapsto C_3& C_2&\mapsto E_3& B_1&\mapsto A_2 \end{align*}
This is a reversal of points, but reversing the order of points in the cross ratio will not change its value. So by now you have fixed $E_3$ by intersecting two lines, and now have shown $T(C_2)$ to be the intersection of $A_2\vee B_2$ with $D_2\vee E_2$. Therefore you now know that $T(C_2)=C_3$. With a similar argument you get $T(B_2)=B_3$. And now you can use these to construct $T(E_1)$:
\begin{align*} T(E_1) &= T\bigl((A_1\vee B_2)\wedge(D_1\vee C_2)\bigr) \\&=\bigl(T(A_1)\vee T(B_2)\bigr)\wedge(T(D_1)\vee T(C_2)\bigr) \\&=(A_2\vee B_3)\wedge(D_2\vee C_3) = E_2 \end{align*}
So at this point you know that the fifth point of the outer pentagram will be mapped to the fifth corner of the inner pentagon, even though only the other four points define that map.
Iterative transformations
So what is the transformation which maps the second to the third pentagram? It is the same: as that transformation preserves incidence, it maps the first pentagram and its inscribed pentagram to the second pentagram and its inscribed pentagram, i.e. the third pentagram. So you are in fact asking about the limit of repeated applications of a single transformation.
Limit case
Now consider the limit case. Your projective transformation $T$ is a $3\times 3$ matrix, and its eigenvectors correspond to the fixed points of the transformation. Consider the diagonal form of that matrix. Repeating the transformation means taking powers of that diagonal matrix. But as we are dealing with homogenous coordinates, you can rescale that matrix arbitrarily. This means that in the long run, only the eigenvalue with the maximal absolute value will have any contribution, the other two eigenvalues will converge to zero after rescaling.
So the homogenous coordinates of the convergence point are given by the eigenvector corresponding to the maximal (absolute) eigenvalue.
Solution 2:
This question was posed a very long time ago, but you may still find this interesting. The construction which you consider was introduced by Richard Schwartz in 1992, and it is called the $\textit{Pentagram Map}$. In his introductory paper, he discusses precisely the phenomenon you observe: pentagons are invariant under this mapping, up to projective equivalence (the proof is a straightforward cross ratio computation). The answer regarding the convergence of this process was provided by Glick in this paper.
The pentagram map has all sorts of curious properties. On hexagons, it is an involution. Even more interestingly, when this map is applied to bigger polygons, it displays quasi-periodic behavior. In fact, it was shown by Ovsienko, Schwartz and Tabachnikov in this paper that the pentagram map is a $\textit{discrete integrable system}$, and it has since been the subject of much research, relating it to the K-dV hierarchy, Poisson structures, cluster algebras, along with many more interesting areas of math, most of which are far beyond me. Anyway, here are a few more papers you may find interesting:
- ''Integrability of the Pentagram Map'' by Soloviev (link)
- ''Pentagram Maps and Refactorization in Poisson-Lie Groups'' by Izosimov and Khesin (link)
- ''The Pentagram Map and Y-Patterns'' by Glick (link)
Hope this helps!