Why isn't $D_\infty$ the set of symmetries of a circle?

What you think of when taking $n$ to $\infty$ is this:

That is, you've got polygons inscribed in a common circle. Note however that even in that model, the limit to infinity would not get the full circle, because only rational multiples of $2\pi$ will appear.

But the limit that leads to $D_\infty$ is more like this:

And if you continue that to infinity, it's easy to see that the the sides will approach a straight line with equidistant points, and a rotation will end up as a translation on that line.

Although polygons "approach" the circle, the circle has quite a few more points than just the union of the vertices of all the $n$-gons inscribing it, and consequently it "should" have more symmetries. The symmetry group (rotations and reflections) of the circle is known as $\text{O}(2, \mathbb{R})$ or sometimes just $\text{O}(2)$, which can be seen as the set of all orthogonal ($A^t = A^{-1}$) 2-by-2 matrices with real entries, under multiplication.

Instead, you could think of $D_\infty$ as being the "limit" of the presentations for the finite dihedral groups, $$D_n = \langle r, s \mid r^n = s^2 = 1,\, srs = r^{-1} \rangle.$$ As $n$ goes to infinity, the statement that $r^n = 1$ simply drops out, because what "$r^n = 1$" is really saying is "$\text{ord}(r) = n$", and we write $\text{ord}(r) = \infty$ to mean $r$ has infinite order. So $$D_\infty = \langle r, s \mid s^2 = 1,\, srs = r^{-1}\rangle$$ which some authors take as the definition of $D_\infty$. Others define the infinite dihedral group as the isometries of $\mathbb{Z}$ as a subset of $\mathbb{R}$, in which case $x \mapsto x + 1$ can be thought of as $r$ (a "rotation") and $x \mapsto -x$ can be thought of as $s$ (a "reflection").

$D_\infty$ is the group of symmetries of an $\infty$-agon, of which a simple model is the set $\mathbb Z$ inside of $\mathbb R$.

Finite dihedral groups, in general, are defined as particular quotients of the free group on two letters, with the associated normal subgroup being generated by the relations $r^n = f^2 = (rf)^2 = 1$. The natural extension of this idea to the infinite case is to let $\langle r \rangle \cong \mathbb{Z}$ since the rotation subgroup is always cyclic, and the size of the rotation subgroup is the only thing differentiating all the various finite dihedral groups, structurally speaking.

One problem with thinking about $D_{\infty}$ as the symmetry group of a circle is that the cardinalities simply don't match up. As a quotient of a free group on two letters, $D_{\infty}$ is countably infinite. On the other hand, the rotation subgroup of the group of symmetries of a circle is uncountably infinite as we can rotate $S^1$ by any angle $\theta \in [0, 2\pi)$.

Note that we can realize $D_{\infty}$ as the symmetry group of the set of integer points on a real number line: the rotation can be thought of as the translation $x \mapsto x \! + \! 1$ and the reflection as reflection across the origin. And this kind of makes sense given that the internal angle of a regular $n$-gon measures $\displaystyle 180- \frac{360}{n}$ degrees, and this approaches $180^{\circ}$ as $n \to \infty$.