Solution 1:

It could be any compound noun of which the head, the "main noun", is not the final part of the compound. This includes all compound nouns whose final parts are not nouns. "Bystanders" is normally not written as you did. In "passers-by", the final part is "by", which is not a noun. Note that there might be some controversy about the correct spelling of some such words, but I just try to be consistent. A few examples:

  • runners-up
  • fins-de-siècle (and many more French words)
  • houses of cards
  • Commanders-in-Chief
  • sons-in-law
  • attorneys at law
  • (tea)spoonsful
  • ...

Solution 2:

Men-o’-war is a nice one: shows that irregular plurals are just as susceptible to this construction.

Also: pickers-up, on the same pattern as passers-by.

Solution 3:

  • courts-martial
  • men-at-arms

And a few more from Wikipedia:

  • bills of attainder
  • directors general
  • fees simple absolute
  • ships of the line
  • ministers-president
  • knights-errant
  • procurators fiscal

Solution 4:

Besides those in others' answers, there are: attorneys general, secretaries general, solicitors general, postmasters general, governors-general, etc.; [noun]s-elect; and Knights Templar and Knights Hospitaller.

Solution 5:

Another example is culs-de-sac, the plural of cul-de-sac.