Cannot get RK4 to solve for position of orbiting body in Python

Physics

The Newton law gives you a second order ODE u''=F(u) with u=[x,y]. Using v=[x',y'] you get the first order system

u' = v
v' = F(u)

which is 4-dimensional and has to be solved using a 4 dimensional state. The only reduction available is to use the Kepler laws which allow to reduce the system to a scalar order one ODE for the angle. But that is not the task here.

But to get the scales correct, for a circular orbit of radius R with angular velocity w one gets the identity w^2*R^3=G*M which implies that the speed along the orbit is w*R=sqrt(G*M/R) and period T=2*pi*sqrt(R^3/(G*M)). With the data given, R ~ 10, w ~ 1, thus G*M ~ 1000 for a close-to-circular orbit, so with M=20 this would require G between 50 and 200, with an orbital period of about 2*pi ~ 6. The time span of 10 could represent one half to about 2 or 3 orbits.

Euler method

You correctly implemented the Euler method to calculate values in the last loop of your code. That it may look un-physical can be because the Euler method continuously increases the orbit, as it moves to the outside of convex trajectories following the tangent. In your implementation this outward spiral can be seen for G=100.

This can be reduced in effect by choosing a smaller step size, such as dt=0.001.

You should select the integration time to be a good part of a full orbit to get a presentable result, with above parameters you get about 2 loops, which is good.

RK4 implementation

You made several errors. Somehow you lost the velocities, the position updates should be based on the velocities.

Then you should have halted at fx(x + .5*kx1, y + .5*kx1, t + .5*dt) to reconsider your approach as that is inconsistent with any naming convention. The consistent, correct variant is

fx(x + .5*kx1, y + .5*ky1, t + .5*dt) 

which shows that you can not decouple the integration of a coupled system, as you need the y updates alongside the x updates. Further, the function values are the accelerations, thus update the velocities. The position updates use the velocities of the current state. Thus the step should start as

 kx1 = dt * fx(x,y,t) # vx update
 mx1 = dt * vx        # x update
 ky1 = dt * fy(x,y,t) # vy update
 my1 = dt * vy        # y update

 kx2 = dt * fx(x + 0.5*mx1, y + 0.5*my1, t + 0.5*dt)
 mx2 = dt * (vx + 0.5*kx1)
 ky2 = dt * fy(x + 0.5*mx1, y + 0.5*my1, t + 0.5*dt)
 my2 = dt * (vy + 0.5*ky1)

etc.

However, as you see, this already starts to become unwieldy. Assemble the state into a vector and use a vector valued function for the system equations

M, G = 20, 100
def orbitsys(u):
     x,y,vx,vy = u
     r = np.hypot(x,y)
     f = G*M/r**3
     return np.array([vx, vy, -f*x, -f*y]);

Then you can use a cook-book implementation of the Euler or Runge-Kutta step

def Eulerstep(f,u,dt): return u+dt*f(u)

def RK4step(f,u,dt):
    k1 = dt*f(u)
    k2 = dt*f(u+0.5*k1)
    k3 = dt*f(u+0.5*k2)
    k4 = dt*f(u+k3)
    return u + (k1+2*k2+2*k3+k4)/6

and combine them into an integration loop

def Eulerintegrate(f, y0, tspan):
    y = np.zeros([len(tspan),len(y0)])
    y[0,:]=y0
    for k in range(1, len(tspan)):
        y[k,:] = Eulerstep(f, y[k-1], tspan[k]-tspan[k-1])
    return y


def RK4integrate(f, y0, tspan):
    y = np.zeros([len(tspan),len(y0)])
    y[0,:]=y0
    for k in range(1, len(tspan)):
        y[k,:] = RK4step(f, y[k-1], tspan[k]-tspan[k-1])
    return y

and invoke them with your given problem

dt = .1
t = np.arange(0,10,dt)
y0 = np.array([10, 0.0, 10, 10])

sol_euler = Eulerintegrate(orbitsys, y0, t)
x,y,vx,vy = sol_euler.T
plt.plot(x,y)

sol_RK4 = RK4integrate(orbitsys, y0, t)
x,y,vx,vy = sol_RK4.T
plt.plot(x,y)

You are not using rkx, rky functions anywhere! There are two return at the end of function definition you should use return [(kx1 + 2*kx2 + 2*kx3 + kx4)/6, (mx1 + 2*mx2 + 2*mx3 + mx4)/6] (as pointed out by @eapetcho). Also, your implementation of Runge-Kutta is not clear to me.

You have dv/dt so you solve for v and then update r accordingly.

for n in range(1,len(t)): #solve using RK4 functions
    vx[n] = vx[n-1] + rkx(vx[n-1],vy[n-1],t[n-1])*dt
    vy[n] = vy[n-1] + rky(vx[n-1],vy[n-1],t[n-1])*dt
    x[n] = x[n-1] + vx[n-1]*dt
    y[n] = y[n-1] + vy[n-1]*dt

Here is my version of the code

import numpy as np

#constants
G=1
M=1
h=0.1

#initiating variables
rt = np.arange(0,10,h)
vx = np.zeros(len(rt))
vy = np.zeros(len(rt))
rx = np.zeros(len(rt))
ry = np.zeros(len(rt))

#initial conditions
vx[0] = 10 #initial x velocity
vy[0] = 10 #initial y velocity
rx[0] = 10 #initial x position
ry[0] = 0 #initial y position

def fx(x,y): #x acceleration
     return -G*M*x/((x**2+y**2)**(3/2))

def fy(x,y): #y acceleration
     return -G*M*y/((x**2+y**2)**(3/2))

def rk4(xj, yj):
    k0 = h*fx(xj, yj)
    l0 = h*fx(xj, yj)

    k1 = h*fx(xj + 0.5*k0 , yj + 0.5*l0)
    l1 = h*fy(xj + 0.5*k0 , yj + 0.5*l0)

    k2 = h*fx(xj + 0.5*k1 , yj + 0.5*l1)
    l2 = h*fy(xj + 0.5*k1 , yj + 0.5*l1)

    k3 = h*fx(xj + k2, yj + l2)
    l3 = h*fy(xj + k2, yj + l2)

    xj1 = xj + (1/6)*(k0 + 2*k1 + 2*k2 + k3)
    yj1 = yj + (1/6)*(l0 + 2*l1 + 2*l2 + l3)
    return (xj1, yj1)

for t in range(1,len(rt)):
    nv = rk4(vx[t-1],vy[t-1])
    [vx[t],vy[t]] = nv
    rx[t] = rx[t-1] + vx[t-1]*h
    ry[t] = ry[t-1] + vy[t-1]*h

I suspect there are issues with fx(x,y,t) and fy(x,y,t)

This is the case, I just checked my code for fx=3 and fy=y and I got a nice trajectory.

Here is the ry vs rx plot: