Stability region for two step Nyström method

Solution 1:

The eigenvalues of $A$ are $hλ\pm\sqrt{1+(hλ)^2}\approx \pm\exp(\pm hλ)$. The method is stable if for $Re(λ)<0$ the iteration converges to zero. As the product of the eigenvalues is always $-1$, you will always have one eigenvalue greater than $1$ in absolute value, and thus the component of the solution corresponding to it growing, not falling to $0$.

In conclusion, the method is nowhere stable.

Following Hairer/Wanner: "Solving ODE II", ch. V.1 "Stability of multi-step methods", the characteristic polynomials here are $\rho(\zeta)=\zeta^2-1$ and $\sigma(\zeta)=2\zeta$ with characteristic equation $\rho(\zeta)-\mu\sigma(\zeta)=0$, where $\mu=hλ$, and $\zeta=e^{hλ}$ is the step factor of the exact solution.

Then the stability domain is defined as the set of all $\mu$ where the characteristic equation only has solutions in the closed unit disk. But again, here the solutions have product $-1$, so they can not all be inside the unit disk, and both roots are on the boundary for $\mu$ on the segment between $-i$ and $i$. This is called "weak instability" and causes the oscillating behavior (slowly increasing from floating point noise) shown in my previous answer resp. vol. I, chapter III.9 of the cited book.