Example of dynamical system where: $NW(f) \not\subset \overline{R(f)}$

Solution 1:

I want to clarify the idea that I wrote in the comments. Suppose you have a planar system with the following behaviour: all trajectories enter some compact domain that contains attracting homoclinic figure-eight. I'll explain below how to construct such system.

We can say that $\omega$-limit points, recurrent and non-wandering points can be only in this compact domain. We have two unstable foci ($\alpha_L$, $\alpha_R$) and saddle $\sigma$ — they are clearly belong to both $NW$ and $R$. Who else can be in $R$? Points not on the homoclinic figure-eight can't be recurrent, because homoclinic figure-eight is attractive. If we consider points on this figure-eight, then they clearly can't be in $R$ — as the time goes to infinity, they go to $\sigma$ no matter what, they don't return to themselves. So the only points which are $\omega$-recurrent in this domain are $\alpha_L$, $\alpha_R$ and $\sigma$. So, the $\overline{R}$ is just $R$ and it's a finite number of points.

However, each point of the homoclinic figure-eight lies in non-wandering set. The proof of this fact is usually based on $\lambda$-lemma. I've tried to illustrate the sketch of the proof for planar system. If you take some point $p$ on the unstable manifold of a saddle (close enough to saddle to apply Grobman-Hartman theorem) and some transversal curve $\gamma$, then after some time $T$ the $f^T{p}$ return back cloes enough to saddle. This point brings small segment of curve $f^{T}(\gamma)$ with it. It is very easy to show using Grobman-Hartman that if we apply flow to this segment $f^{T}(\gamma)$, then eventually the image of this segment will intersect segment of curve $\gamma$ close to initial point $p$. Thus we always have points in some small neighbourhood of $p$ that return close enough to $p$ and because of that $p$ is non-wandering point. Because each point on single homoclinic loop is a $f^\tau (p)$ for some $\tau$, we proved that single homoclinic loop consists of non-wandering points. The same applies to another homoclinic loop, thus homoclinic figure-eight belongs to $NW$. This already shows that $\overline{R} \subsetneq NW$.

How to construct particular example of such system?

If you want to have particular example of such system, you can construct it this way. Take Duffing oscillator without friction ($\dot{x} = P_D(x, y), \; \dot{y} = Q_D(x, y)$). It is a Hamiltonian system with Hamiltonian $H(x, y)$ which has a critical level set $C_{\rm crit}$ that contains a saddle equilibrium with its separatrices forming a homoclinic figure-eight. Now perturb this vector field this way: $$ \dot{x} = P_D(x, y) - \alpha (H(x, y) - C_{\rm crit}) \cdot Q_D(x, y), $$ $$ \dot{x} = Q_D(x, y) + \alpha (H(x, y) - C_{\rm crit}) \cdot P_D(x, y). $$

Geometrically this transformation stretches the initial vector field and rotates it a little bit depending on the sign of $\alpha \Bigl ( H(x, y) - C_{\rm crit} \Bigr )$. Note that vector field remains unchanged at equilibria and at homoclinic figure-eight so they are preserved in the perturbed system. How other trajectories behave? Let's check how $H(x, y)$ changes in time for perturbed system:

$$ \frac{d H(x, y)}{dt} = \frac{\partial H}{\partial x} \dot{x} + \frac{\partial H}{\partial y} \dot{y} = \frac{\partial H}{\partial x} \Bigl(P-\alpha (H(x, y) - C_{\rm crit}) Q \Bigr) + \frac{\partial H}{\partial y} \Bigl (Q+\alpha (H(x, y) - C_{\rm crit}) P \Bigr) $$

Since the original system was Hamiltonian, then $P = \frac{\partial H}{\partial y}$, $Q = - \frac{\partial H}{\partial x}$ and everything boils down to

$$ \frac{\partial H}{\partial x} \Biggl (\frac{\partial H}{\partial y}+\alpha (H(x, y) - C_{\rm crit}) \frac{\partial H}{\partial x} \Biggr ) + \frac{\partial H}{\partial y} \Biggl(- \frac{\partial H}{\partial x}+\alpha (H(x, y) - C_{\rm crit}) \frac{\partial H}{\partial y} \Biggr ) = \alpha \Bigl (H(x, y) - C_{\rm crit} \Bigr ) \Biggl (\Bigl (\frac{\partial H}{\partial x} \Bigr )^2 + \Bigl (\frac{\partial H}{\partial y} \Bigr )^2 \Biggr ). $$

So, by appropriate choice of $\alpha$ we can say that $H(x, y)$ becomes a sort of Lyapunov function for homoclinic figure-eight of perturbed system. Of course this is strictly because of how level sets of this Hamiltonian look like and how we perturb this system.