Remote Desktop - Does leaving a session open slow down the server?

Will it slow the server down? Possibly. A disconnected session still consumes resources.

The larger implication is the potential security ramifications of leaving an unattended session running on the server. For that reason alone I would strongly encourage the practice of logging out of the session.