Is it correct to say "mammals became the dominant species"?

Solution 1:

You could say "mammalian species became dominant". But you don't need to. 'Species' as a plural agrees with 'mammals'. Some audiences may disagree for technical or factual reasons, but they won't be confused.

Solution 2:

Gold star for finding a situation where a noun with identical plural and singular forms rescues a biology error. But "dominant species" seems to be a familiar phrase, even in this misapplication (https://en.wikipedia.org/wiki/Dominant_Species).

Readers who are confused by the phrase "dominant class," as a phrase: likely few.

Readers who are confused by the phrase "dominant species," as a phrase: likely few.

Readers who read "mammals became the dominant species," are concerned by their existing knowledge that they are in fact a class, but are reassured upon remembering that "species" is also plural: likely few.

Also, a footnote: although interpreting "species" as a plural is a clever way of rescuing the grammaticality, it might be factually incorrect. "Mammals became the dominant class" is not the same thing as "Some or all of the mammalian species became the dominant species in their ecosystem."