How do I test if a player moved a certain distance from an area in minecraft?

You could define that area using coordinates and volume in the target selector of a command that adds a scoreboard tag. Anyone without that tag would then be outside that range. The only downside in your case is that the defined area will not be circular but square.

This would use two command blocks in a chain. The first removes the tag from all players. The second adds the tag only to players inside the area. These command blocks need to remain loaded so I recommend placing them in the spawn chunk.

I also recommend preventing command block output to prevent OP players from having their chat filled with command block spam. Use command:

/gamerule commandBlockOutput false

First command block is Repeat Unconditional Always Active and it's command:

scoreboard players tag @a remove InArea 

Second command block is Chain Unconditional Always Active and it's command:

scoreboard players tag @a[x=X,y=Y,z=Z,dx=DX,dy=DY,dz=DZ] add InArea 

The two sets of coordinates in the second command target selector are two opposite corners of the area. In your case, Y could be 1 and DY could be 256.

Now you can target anyone without that tag: @a[tag=!InArea]

As an example, lets make them all say hello:

/execute @a[tag=!InArea] ~ ~ ~ say Hello