Why is it that score is used in singular when referring to several groups of 20?

Solution 1:

The word score follows the same rule that applies for the words dozen, hundred, thousand and million. When they are used with a number to denote exact quantity, their form doesn't change, e.g. we say two hundred, ten thousand, three dozen, five million.

The plural form with these words is used when the number denoted isn't exact and we just want to emphasise the fact that there are many items, e.g. hundreds of years ago, dozens of times, etc.

Solution 2:

It's the same reason we don't say "Two hundreds dollars". The number, whether twenty or eleven thousand three hundred, refers to the years, not to the 'groups of years'. Nobody would say *four tens years; you can say four decades, precisely because a decade is ten years. But a score is just twenty and cannot (in this sense) be pluralized.