Can spammers extract valid email addresses from my mail server?

It's possible, depending on the specifics of your mail server and it's configuration. Directory harvesting comes to mind.