Are compact complete geodesics closed?

Let $(M,g)$ be a compact Riemannian manifold. Is there an example of a geodesic $c:\mathbb{R}\to M$ s.t. $c(\mathbb{R})$ is compact, $c$ is NOT periodic (i.e. be NOT a closed geodesic) ?

(I'm assuming you mean a smooth Riemannian manifold.) I really struggled with this problem. Here is my outline of an attempt.

First consider this as a special case of the following:

Theorem. If $X$ is a smooth vector field on a manifold $N$, and $f:\mathbb R \to N$ is a solution to the differential equation $\dot f(t) = Xf(t)$, then either $f(t_1) = f(t_2)$ for some $t_1 \ne t_2$ (and hence $f$ is periodic or constant), or the image $f(\mathbb R)$ is not compact.

How to prove the theorem? $X$ defines a group action $T$ by $\mathbb R$ onto $N$ where $T(t)(p) = f(t)$, with $f$ solving the ODE $\dot f(t) = Xf(t)$, $f(0) = p$. Now if $f(\mathbb R)$ is compact but not periodic or constant, then there must be one point on $f(\mathbb R)$ that is a limit point. But since the action $T$ is continuous, and transitive on $f(\mathbb R)$, it follows that $f(\mathbb R)$ is a perfect subset of $N$.

Find a codimension $1$ submanifold $B$ (not compact, eg like a small open subset of a plane in $\mathbb R^3$) in a neighborhood of $f(0)$ that is transversal to the flow $X$, that is, $X$ is never parallel to $TB$. Persuade yourself that the closure of $B$ intersect $f(\mathbb R)$ must be a perfect set. But $f(\mathbb R)$ can only hit $B$ countably many times. And a perfect set must be uncountable.

Now to answer your question: apply the above to the vector field $X$ on $N = TM$ that on any chart takes $(p,v)$ to $(v,\phi(p,v))$, where $\ddot p = \phi(p,\dot p)$ is the usual Euler-Lagrange equation that defines a geodesic. So the image $f(\mathbb R) \subset TM$ is not compact. Suppose for a contradiction that $\pi(f(\mathbb R))$ is compact where $\pi:TM \to M$ is the canonical projection.

Here I am stuck!

The answer is, as expected no. What follows is more of a proof sketch than a complete proof. Contradiction is attained in the same fashion as in Stephen M.S.'s answer. We make the obvious assumption $\dim M\geq 2$.

Consider some sequence of real numbers $r_n$ that tends to infinity. By compactness of $c(\Bbb R)$, there is some subsequence, say $s_n$, and some real number $t_0$, such that $c(s_n)\to c(t_0)$. Actually, by compactness of the unit sphere, we may arrange $\dot c(s_n)\to\dot c(t_0)$.

We need to weed out a little annoyance: the point $c(t_0)$ may be reached infinitely often. We can safely assume this is not the case. To convince ourselves, let us consider the map $$c\times c:\Bbb R\times\Bbb R\setminus\Delta\to M\times M,\quad(s,t)\mapsto(c(s),c(t))$$ it has constant rank ($2$, since $\dim M\geq 2$), and is easily seen to be transveral to the diagonal $\Delta_M\subset M\times M$ (otherwise the geodesic would be periodic). Hence the times $t\neq t'$ such that $c(t)=c(t')$ are isolated in $\Bbb R^2$, in the sense that $(c\times c)^{-1}(\Delta_M)$ is a discrete, hence countable, subset of $\Bbb R\times\Bbb R\setminus\Delta$ .

Thus, for some small $\epsilon>0$, the point $c(t_0+\epsilon)$ is only ever attained once by the curve $c$. Also, the fact that the derivatives $\dot c(s_n)$ converge to $\dot c(t_0)$ readily imply that the derivatives $\dot c(s_n+\epsilon)$ converge to $\dot c(t_0+\epsilon)$. Finally, modulo an affine change of parameter $t\mapsto t+a$ we may assume

For all $t\neq 0$, $c(t)\neq c(0)$, and there is a seqence $t_n\to+\infty$ such that $\dot c(t_n)\to\dot c(0)$.

In any case, if you take a little (compact) piece of hypersurface $H$ that contains $c(0)$ and is (say) orthogonal to $\dot c(0)$, we see that (by construction of the sequence $t_n$) the curve $c(t)$ will traverse $H$ infinitely often.

Hence, the set $S\subset\Bbb R$ of $t$ such that for some compact hypersurface patch $H$ containing $c(t)$ and transversal to $\dot c(t)$, $c(t)$ is non-isolated in $H\cap c(\Bbb R)$ is non-empty, since it contains $0$.

$S$ is easily seen to be open in $\Bbb R$. This necessitates the hypothesis that $\dot c(t_n)\to\dot c(0)$. But more is true: compactness of $M$ implies that there is a uniform $r>0$ such that

1. $\forall p\in M,\exp_p:B(0,r)\subset T_pM\to M$ is a diffeomorphism,
2. $\forall p\in M$, the pullback metric $\exp_p^*g$ on the $r$-ball centered at $0$ of $T_pM$ deviates less than some small uniform $\delta>0$ from $g_p$.

This will imply that there exists some positive $\epsilon>0$ such that for all $t\in S$, $(t-\epsilon,t+\epsilon)\subset S$. Since $S$ is non-empty, this implies $S=\Bbb R$

(EDIT. This last point is not true as stands: There are problems with points that are visited infinitely many times. This invalidates the proof.)

As a consequence, the intersection we were considering earlier $$H\cap c(R)$$ is closed and without isolated points (up until now, we only knew $c(0)$ was not isolated, but $S=\Bbb R$ implies that no point in this intersection is isolated). But it is also countable by some transversality argument (which may require one to slightly perturb the piece of hypersurface $H$). Hence it is a countable perfect set, of which there can be none in a manifold.