Which is correct, criterion- or criteria- in a compound adjective?

I would prefer "criterion-based" (just as I prefer "rule-based" to "rules-based"). To me, there doesn't seem to be any good reason to prefer "criteria-based": I think the context makes it fairly obvious that "criterion-based" could refer to something based on more than one criterion. But other people might have different opinions. I'm talking about "preferences" and "opinions" because if "criteria-based" is used by good writers in the relevant field, it seems pedantic to call it "incorrect".

"Criterion-based" seems to be more common than "criteria-based" in American English

I searched the Corpus of Contemporary American English (COCA) and found "criterion-based" to have a frequency of 41, vs. "criteria-based" which only had a frequency of 7. All 41 examples of "criterion-based" came from an academic context, so it's not just a colloquialism.

I took a look at the collocates, and the only one where criteria-based was more common than criterion-based in the COCA corpus was with the word "selection", and that was only based on four examples of criteria-based used with "selection" vs. no examples of criterion-based used with "selection". (Google Books provides a number of examples of "criterion-based selection", so it isn't actually nonexistent.) With all other common collocates (such as "was", "strategies", "content", "assessment", "development", "performance", "test"), criterion-based was more common. My overall intepretation of this data would be that, in American English, criterion-based is more common than criteria-based in academic contexts; there are no clear differences in meaning and no contexts where the latter is clearly preferred.

(I haven't been able to get data that I trust from the Google Ngram Viewer because I'm somewhat uncertain about when it's interpreting the character - as a hyphen vs. as a minus sign. Regardless, none of the charts I saw when I was fiddling around with it showed "criteria-based" as being more frequent than "criterion-based" in either American or British English.)

Plural forms can be used in compound words; it's uncommon but not always incorrect

As you observe, nouns used as the first element of compounds in English are often invariant for grammatical number, taking the form of the singular. This is evident not only in the formation of compound adjectives, but also in the formation of compound nouns, or the use of the "attributive noun" construction (a noun placed before and used to modify the meaning of another noun).

However, it is not absolutely impossible for an attributive noun, or for a noun serving as the first element of a compound adjective, to be in the plural form.

The most uncontroversial examples are probably certain pluralia tantum where singularizing the word would give another word with a different meaning. Clothes are hung on a "clothes line", not on a "cloth line"; glasses are stored in a "glasses case" (or "eyeglass case"), not in a "glass case".

There is also a tendency, although not as strong, to use plural forms attributively for some other pluralia tantum. The singular form of "scissors" has no other meaning, and we do see "scissor" used as an attributive form or in compound adjectives, but some people use "scissors" in these contexts. So, there is variation between "scissors kick" and "scissor kick". Likewise, there is some variation between "pant leg" and "pants leg".

Shosht, in the comments, brought up another example that seems particularly relevant: "results-based" ("result-based" also exists, but seems to be less common).

Speakers sometimes allow things with irregular plurals that aren't normally allowed with regular plurals

There is apparently also an effect whereby irregular plural forms are more tolerated than regular plural forms as attributive nouns, although it doesn't seem overwhelmingly strong to me (see the examples "teeth-marks" and "mice-infested" in the following question: Irregular plurals in noun adjuncts, and the references to the literature discussing these examples).

A similar question, Grammatical number of Latin nouns used attributively before other nouns, provides the examples of multistrata cakes and multimedia ___.

I also think it's relevant to mention that, from a certain pedantic point of view, it could be considered "incorrect" to use any of the following words as singular nouns, since they are plural in Latin: agenda, data, stamina, media, insignia. People often disregard this and use them as singular anyway. (The specific patterns and histories of singular usage are different for each word, but my point is that rules can only tell you so much about matters of "correctness" for this kind of thing.) It seems likely to me that this is a large part of the explanation for the use of "data set" instead of "datum set", but that could also possibly be compared to the compound that you asked about. The use of "criteria" as a singular noun is currently not very common and not generally accepted, but it is apparently common enough at least for some sources to warn against this usage (see "criteria / criterion", Common Errors in English Usage), so there might be some small connection, at least for some people, between the nonstandard use of "criteria" as a singular noun and the use of "criteria-based". (For more on the grammatical number of "criteria", see “Criteria” versus “criterion”)