Irregular plurals in noun adjuncts
It's not just you. Choosing a more common existing collocation, namely woman doctor, and consulting Google Ngram Viewer, we find that "women doctors" is much more common than "woman doctors", but that both exist. And since lady doctor is also common (albeit much less so than woman doctor), we have a good control group — "lady doctors" is attested, but *"ladies doctors" is not (at least in the Google Ngram corpus).
Here's a snapshot:
[source]