Why call them infinite/finite verbs?
Solution 1:
According to Etymonline, the derivation is:
infinitive
"simple, uninflected form of a verb," 1510s, from L.L. infinitivus "unlimited, indefinite," from L. infinitus (see infinite). "Indefinite" because not having definite person or number.
So it is used more of the sense of indefinite. It doesn't mean something that approaches the infinite in scope, at least not in the mathematical sense.