Add players to team if they are within range?
The @s selector refers to the entity which is executing that command. Since the command is being executed by a command block, @s is referring to the command block and that's why it's not working.
There is a much easier way of doing this which doesn't require you to test for anything.
To add someone within range to a team: /team join someTeam @a[distance=..5]
And to remove players from the team if they are out of range: /team leave @a[team=someTeam,distance=5..]