Stability when eigenvalues are zero : $x' = -x + y + x^2 + ax^3, \space y' = x - y + x^2 + bxy + y^3$

When there are some zero eigenvalues and no positive eigenvalues of the Jacobian, the stability is that of the flow along the center manifold. So what you to do (and this is the canon for this type of problems) is finding the equation on the center manifold (more precisely, on any center manifold in case it is not unique).

How do you do that? First pass to coordinates $$u=x+y,\quad v=x-y$$ (their axes are tangent to the stable and center spaces). After writing the equation in the form $$u'=P(u,v),\quad v'=Q(u,v),$$ the second equation along the center manifold will give the stability (which doesn't change with the change of variables). We start like this: write the graph of the center manifold as $$u=cv^2+dv^3+\cdots,$$ substitute in the first equation, deduce from the equation $$(cv^2+dv^3+\cdots)'=P(cv^2+dv^3+\cdots,v)$$ what are $c,d,\ldots$ (just equate equal powers) and replace that in the second equation $$v'=Q(cv^2+dv^3+\cdots,v)$$ now already the constants $c,d,\ldots$ that you determined. The stability of the origin in this equation will be the stability of the origin in the original equation.