How to select entities not within a given cylinder in minecraft

To expand on Fabian's first solution i bring example of how to implement it.

First of we need to setup a scoreboard objective

/scoreboard objectives add inCylinder dummy inCylinder

By using this command we create a scoreboard objective InCylinder that can hold any number,

Then we summon 1 armor stand with coordinate 0 on y axis, so

/summon armor_stand 413 0 456 {NoGravity:1b}

413 is the x coordinate of the center of the cylinder,

456 is the z coordinate of the center of the cylinder,

0 shouldn't be changed

next step is to summon armor stands recursively above the original armor stand, Backup your world before doing this step

/execute @e[type=armor_stand,c=-1] ~ ~ ~ summon armor_stand ~ ~1 ~ {NoGravity:1b}

you put this in a repeating command block and let it run for ~14 seconds to summon armor stands up to y=256 because you can't build above 256 it would be useless to summon armor stands

now for the actual detecting

execute @e[type=armor_stand] ~ ~ ~ execute @a[r={SPECIFY_RADIUS_HERE}] ~ ~ ~ scoreboard players set @s inCylinder 2

put the above command into a repeating command block, and set it to always active, then connect it to a chain command block (always active)

execute @a ~ ~ ~ scoreboard players remove @s inCylinder 1

then the last chain command block does

execute @a[score_inCylinder=0] ~ ~ ~ {YOUR_COMMAND_HERE}

I don't think this is exactly possible. The three types of area related target selector arguments are x/y/z, dx/dy/dz and r/rm. x/y/z moves the starting point, dx/dy/dz form a cuboid and r/rm forms a sphere. If you combine the two, you can only select the cross area of the two. This can never form a cylinder.

But you can approximate it. You want to target players flying out anyway, so with the maximum speed in Spectator you won't detect them at exactly the border anyway (because command blocks/functions are restricted to 20/s and you can move up to 4 blocks in that time in Spectator mode. So you don't need to be perfectly exact with it anyway. Here are two solutions for an approximation:

Solution 1, approximating the mantle: You can put armor stands along the center axis of the cylinder (every 10 blocks should be enough), then give every player inside a radius around them a tag and then teleport everyone without the tag. Don't forget to remove the tag from everyone (*) afterwards! This makes your space look a bit like a pyramid cake (without the hole in the middle):

Solution 2, approximating the area: You can use multiple rectangles to approximate the circle. You can either do it like it's done in limit calculations:

or use the rectangles you get here (drag the point), that would probably lower the amount of rectangles you need for the same accuracy. Again, you would tag everyone inside with dx/dy/dz (y=0,dy=256), then teleport everyone without a tag and then remove the tag. This way your area would look like when someone builds a circle in Minecraft out of blocks, but you can choose the resolution.