Why are certain single word compound nouns pluralized in the middle

Solution 1:

In English there's no such thing as a plural form for an adjective or a preposition. So if you have a compound noun, like "court martial", to make it plural you should pluralize the noun and not the adjective. Hence, "courts martial". There aren't two martials; there are two courts.

Similarly, "passersby" make sense because there are two "passers", not two "byes". "Two spoonsful" because there are two spoons that are full, not two fulls that are spoon. Etc.

But "flyby" is in a different category. It is a noun formed by combining a verb and a preposition. There are not two "byes", but neither are there two "flies". Similarly with "takeover" and "shutout". There is no "inner noun" in these cases to pluralize. So it is the combined word as a whole that is a noun, and it makes sense to pluralize it as a whole word and not as a compound.

Solution 2:

According to Warriner's English Grammar and Composition (1977), most plurals of compound nouns are formed in accordance with two rules. First:

The plural of compound nouns written as one word is formed by adding s or es.

EXAMPLES: spoonfuls, cupfuls, leftovers, strongboxes

Second:

The plural of compound nouns consisting of a noun plus a modifier is formed by making the modified word plural.

To determine the word that is modified in a compound word, make each of the parts plural. The modified word is the one that tells what the entire compound is or does. Thus the plural of notary public is notaries public (they are notaries not publics); the plural of mother-in-law is mothers-in-law (they are mothers, not laws); etc.

EXAMPLES: runners-up, editors in chief, lieutenant governors, poets laureate

However, this book does note some exceptions:

The plural of a few compound nouns is formed in irregular ways.

EXAMPLES: drive-ins, stand-bys, six-year-olds, tie-ups

At least in the examples that Warriner's gives, all of the exceptions involve hyphenated compounds. The Chicago Manual of Style and Words Into Type concur with Warriner's on the general rules, but they don't address the exceptions.

Of course, as language marches on, hyphenated forms sometimes become closed-up forms, with the result that a plural that used to follow the Warriner's rule (like passers-by) suddenly becomes an exception to the rule (passersby). In the instance that bib asks about, a look at dictionaries in Merriam-Webster's Collegiate Dictionary series reveals that MW listed passers-by as the primary spelling up through the Sixth Collegiate Dictionary (1943), but that passersby took over in the Seventh Collegiate (1963) and has remained MW's primary spelling ever since.