What is the formal plural of the word theorem?
Solution 1:
It (strictly in Greek or Latin) would be theoremata, just like schemata. This is the general declension for several -ma words originating in Greek.
But this seems artificial, and in any case why would you want to use the Greek plural pattern when the English one does the job?
Solution 2:
As noted in the other answers, "theorems" is the standard usage in math papers. In an odd contrast, however, "lemmata" still has some currency as the plural of "lemma" (an intermediate theorem), and crops up pretty regularly even in very recent papers. On the other hand, my (unresearched) impression is that this too has become less widespread in recent decades.
For comparison, a search for "lemmata" in the text of MathSciNet reviews gives 313 matches, while "theoremata" appears only six times, all in references to the Latin titles of very old papers.
Solution 3:
The accepted plural seems to be theorems. At least big dictionaries such as Collins or American Heritage Dic. give no other plural form. For some Latin or Greek word Latin and Greek plural-endings can be found but obviously not for all Latin/Greek words. That would be an over-academization of the vocabulary. And I think it is a good thing that foreign plural-endings are restricted to a limited number of words.