System with a Lyapunov function over $\mathbb{R}^n$ but not globally asymptotically stable
Solution 1:
Interesting question!
The issue is that some trajectories may tend to infinity.
I don't think I've ever seen an explicit example, but (with some effort!) I managed to come up with one myself. Take $$ V(x,y) = \frac{x^2}{1+x^2} + y^2 , $$ which is clearly positive definite, but doesn't tend to infinity as $\sqrt{x^2+y^2}\to \infty$ (indeed, $V(x,0)\to 1$ as $x \to \pm\infty$).
The level set $\{ V(x,y)=1 \}$ is given by $y=\pm 1/\sqrt{1+x^2}$, a pair of curves which extend out to infinity, approaching the $x$ axis. Between these two curves, you have level sets $\{ V(x,y)=c \}$ for $c \in (0,1)$ which are closed curves encircling the origin, and $\{ V(x,y)=0 \}$ which is just the origin.
Now consider the system $$ \dot x = x \, \frac{3x^2y^2-1}{x^2 y^2+1} ,\qquad \dot y = -y . $$ The idea here is that on the curve $xy=1$ (which for large positive $x$ lies just above the level curve $y=1/\sqrt{1+x^2}$) the system becomes $\dot x=x$, $\dot y=-y$, which has a solution that stays on that curve. That is, $$ x(t)=e^t ,\qquad y(t)=e^{-t} $$ is a particular solution of our system, and it doesn't tend to the equilibrium $(x,y)=(0,0)$, so the system is not globally asymptotically stable.
And on the other hand, the system becomes $\dot x \approx -x$, $\dot y=-y$ when we are close to the origin, so it's locally asymptotically stable.
It remains to show that $\dot V$ is really negative definite: $$ \begin{split} \dot V & = \frac{\partial V}{\partial x} \, \dot x + \frac{\partial V}{\partial y} \, \dot y \\ & = \frac{2x}{(1+x^2)^2} \, x \, \frac{3x^2 y^2-1}{x^2 y^2+1} + 2y \, (-y) \\ & = \frac{-2}{(1+x^2)^2 (1+x^2 y^2)} \biggl( x^2 (1-3x^2 y^2) + y^2 (1+x^2)^2 (1+x^2 y^2) \biggr) \\ & = \frac{-2}{(1+x^2)^2 (1+x^2 y^2)} \biggl( x^2 - 2 x^4 y^2 + x^6 y^4 + y^2 + 2 x^2 y^2 + x^2 y^4 + 2 x^4 y^4 \biggr)\\ & = \frac{-2}{(1+x^2)^2 (1+x^2 y^2)} \biggl( x^2 (1 - x^2 y^2)^2 + y^2 (1+2x^2)(1+x^2 y^2) \biggr) , \end{split} $$ which is clearly negative away from the origin. Done!
(Remark: The rewriting of $\dot V$ in the last step is due to a helpful comment by @SampleTime, which simplified the argument greatly. See the edit history for my own original version, which was much uglier!)
Solution 2:
Just for the record, here's another example that I just found on p. 109 of the book Stability of Motion by Wolfgang Hahn (1967), where it is credited to a 1952 paper in Russian by Barbashin & Krasovskii: $$ \dot x = -\frac{6x}{(1+x^2)^2} + 2y ,\qquad \dot y = -\frac{2(x+y)}{(1+x^2)^2}. $$ With the same(!) $V$ as in the example I came up with in my other answer, $$ V(x,y) = \frac{x^2}{1+x^2} + y^2 , $$ one computes $$ \dot V = -\frac{4}{(1+x^2)^2} \left( \frac{3x^2}{(1+x^2)^2} + y^2 \right) , $$ so $V$ is positive definite and $\dot V$ is negative definite, and hence $V$ is a strong Lyapunov function on all of $\mathbf{R}^2$.
But Hahn shows that trajectories of this system cannot cross the curve $$ y=\frac{2}{x-\sqrt2} \quad (x>\sqrt2) $$ in the direction towards the origin. (He compares $\dot y/\dot x$ to the slope of the curve.)
Hence not all solutions tend to the origin.