Inversion with "many times" at the beginning of a sentence

Solution 1:

Subject-Auxiliary Inversion (with Do-Support) only happens with some adverbs.
Most adverbs don't indulge.
In particular, neither

• *Often do I see him
  nor

• *Many times have I seen him

are normal colloquial American English.

Adverbs can appear almost anywhere in a sentence, especially its beginning and end
(though different types of adverbs have different affordances),
and they rarely affect the word order of the rest of the sentence, no matter where they wind up.

One place where this does happen, though, is with negative adverbs;
but only with certain kinds of negative adverbs. So,

• Rarely do we see him around here any more
  and
• At no time did he break a window.

are OK (though they seem rather old-fashioned to some),
but

• *For no reason did he break a window
  and
• *With no tools did he open a window

are ungrammatical with inversion, though they're fine without it.