Why use a Display manager?
They handle pretty much everything (that isn't displaying graphics on hardware) for X. Logins, greeters, pass X the configuration for external connections, session choice.
You don't need a heavy DM but I would argue that GDM isn't that heavy. LightDM certainly isn't. But if you're really trying to hypermile things, take a look at NoDM. It's very raw and it's only intended for single-user systems. It won't provide you with any security and it might undermine what is already there.
I use NoDM on our media-centre, which just starts up X and then launches Boxee.