"execute" command not working in command block?

Solution 1:

You cannot use tilde (~) as part of a selector argument, unless you're using Bedrock Edition. Since you didn't state that you were using BE, I'm going to assume you're using the standard Java edition.

Your only real option in 1.12 is to use the actual coordinates that you're trying to point to, such as

execute @p[x=40,y=80,z=120,r=5] ~ ~ ~ spawnpoint @s ~3 ~3 ~-5

Or you can shift the execution point with execute, then you can even simplify the rest of the command:

execute @s ~5 ~2 ~17 spawnpoint @p[r=5] ~3 ~3 ~-5

Also notice that I'm using the @s selector for the first spawnpoint command. This will target the player found in the execute portion.
The same applies to the second execute: You want to execute relative to the player who executes the command, so you can use @s.