How should I use hyphenation in the following case?
Grammarly has a brief article describing the use of hyphens in English. In part, it says the following:
Hyphen with Compound Modifiers: Two-Word Adjectives Before Nouns
A compound modifier is made up of two words that work together to function like one adjective. When you connect words with the hyphen, you make it clear to readers that the words work together as a unit of meaning.
If your reviewer is following this or similar advice, then your examples (1) and (2A) are correct:
(1) The evaluation of such averages might be cumbersome, if the tensor order is high.
(2A) Functions only up to tensor-order $r$ need to be considered in the corresponding integrals.
(There is no need to hyphenate "tensor-order" consistently throughout your article for this to work.)
Despite these guidelines, I had no difficulty understanding your initial example without the hyphen, and as a general (non-technical) editor I would not have imposed this "correction."