Trapping Region for ODE System.

Solution 1:

You have a small mistake for your time derivative of the Lyapunov function, namely for $V(x,y)=2 x^2+y^2$ one gets $16 x^2 y^2$ instead of $20 x^2 y^2$, so

\begin{align} \dot{V}(x,y) &= 8(x^2+y^2)−8(x^4+2 x^2 y^2+y^4), \\ &= 8(x^2+y^2)-8(x^2+y^2)^2, \\ &= 8(x^2+y^2)(1-x^2-y^2). \end{align}

Now you want to find $\alpha$ and $\beta$ (with $0 < \alpha < \beta$) such that for the set $A=\{x,y \mid V(x,y) = \alpha\}$ it holds that $\dot{V}(x,y)>0$ and for the set $B=\{x,y \mid V(x,y) = \beta\}$ it holds that $\dot{V}(x,y)<0$. Namely, this would mean that all initial conditions on the level set $A$ of the Lyapunov function would flow towards higher level sets and that all initial conditions on the level set $B$ of the Lyapunov function would flow towards lower level sets. Thus all initial conditions starting on either $A$ or $B$ should end up in some region in-between $A$ and $B$.

It is worth noting that the level sets of $V(x,y)$ do not overlap with levels sets of $\dot{V}(x,y)$. So for $x^2+y^2=1$, where $\dot{V}(x,y)=0$, does not correspond with a level set of $V(x,y)$ and thus $x^2+y^2=1$ would not be a limit cycle.