Is *race* a synonym of *species* or is just a common mistake?
Solution 1:
No. In its entry for race the ‘Oxford English Dictionary's ’ first definition is 'A group of people, animals, or plants, connected by common descent or origin.'
Solution 2:
It all comes down to sex - can they breed?
So you would probably have to troll the murkier corners of fan fiction to find sufficent examples of man-on-dwarf action to determine if they can interbreed.
Species is becoming a less useful term in biology with more genetic information and it's definition is now a bit fuzzy. Race is a good literary alternative.
Solution 3:
At one level, the answer to your question "shouldn't [they] use 'species' instead?" is "Why should they use a technical, scientific word rather than an ordinary English word?".
Looking at it a different way, the set of circumstances we're talking about is the interaction of humans with something that's like humans but not quite human. In the present day, that is a purely fictional scenario - but in earlier centuries it was a reality, when Europeans encountered people who did not look or behave like them. Whether or not these others were of the same species or not was not apparent (and I'm not sure that the question would have been intelligible to many at the time) but it was immediately obvious that they were of different "race".
In time some scholars certainly decided that they were of different species (see for example Samuel Morton) but the divisions have always been referred to as "races" - though as Barrie says, the word has other meanings as well.
From our modern perspective, we know that all extant humans are of one species, so any subdivision can only be races (though nowadays such subdivisions are not regarded as meaningful by most scientiests). But I suggest that the experience of European explorers who first encountered different kinds of humans was very much akin to that of Terran explorers in Science Fiction who make the first contact with humanoid aliens.
Solution 4:
The dictionary also defines 'literally,' as meaning 'literally,' and 'virtually.'
Dictionaries and common usage can only take us so far.
Scientific definitions are much more useful. In SCIENCE, species are defined by speciation. If two individuals can produce viable offspring, they are of the same species.
This is a ricketey definition, as life itself is largely misunderstood. Science struggles to issue a definition even there.
However, calling aliens a different race is merely yet another ideological tool in the belt of racists. Some use it unknowingly, even innocently, but that does not stop their actions from hurting others.
When black kids see the newest Hollywood movie about an alien 'race,' they understand that the language is built against them. It perpetuates a feeling of alienation.
In short: races are like breeds of dogs. Maybe our ears are bigger or shorter, but we're all the same species.
Aliens are a different species.
If Fantasy settings were subjected to real-world scientific rigor (not necessary at all - indeed I believe such rigor would make the setting sci-fi with fantasy trappings) then Orcs, Humans, and Elves (if they could produce Half-Elves, Half-Orcs, etc.) would all be of the same species. However, if each has an origin story involving magical deities, or being shaped out of mud or emerging from a blooming flower, then all bets are off.