Why aren't the enderman spawning in the only possible location?
I've tried going away and coming back
How much? Keep in mind that mobs follow this rule to spawn:
Since you only have a 2x2 spawnable area, it's easy to miss the correct distance to make them spawn. You need to be at least 24 blocks away, but if you exceed 32 blocks they can still despawn.