The origin of 'hmm'

Solution 1:

The OED defines the word prounounced "hmm" as hum, an interjection.

An inarticulate exclamation uttered with the lips closed, either in a pause of hesitation or embarrassment, or as expressing slight dissatisfaction, dissent, etc. (Cf. hem int., h'm int., um int.)

The earliest attestations of this entry are from Shakespeare, but we're going to get back to the cross-references shortly.

I cried hum..But markt him not a word.

• 1598 Shakespeare Henry IV, Pt. 1 iii. i. 154

This would have been one way of applying a vocal tic to paper. The actual vocalization "hmm" is probably much older than the dates offered by OED. It's possible that Shakespeare put the vocalization to paper for the exact purpose of making this pun.

Looking at the related entries brings us to hem.

An interjectional utterance like a slight half cough, used to attract attention, give warning, or express doubt or hesitation. Also used to represent the slight clearing of the throat of a hesitating or non-plussed speaker.

This word has slightly earlier citations, and might be a factor in the development of the entry for "hum" mentioned.

H'm and hm are not attested until much later.

The etymology of the word hum as a noun or verb, as opposed to an interjection, which has a different pronunciation and meaning, might be relevant as well to the evolution of "hmm."

This is attested earliest in written form, and the OED offers this definition of a verb, the earliest form.

To make a low continuous murmuring sound or note, as a bee or other insect; also said of a top or wheel in rapid rotation, a bell vibrating after being struck, etc.

These are the etymology notes:

Known from end of 14th cent.; echoic; compare Middle High German hummen , modern German dialect hummen , hommen , early modern Dutch (Kilian) hummen = hemmen to hem, emit voice

Conclusion

The word is definitely onomatopoeic, as mentioned in other answers. It might be relevant to know that it is possibly a lexicalization of the sound we use to clear our throats, as suggested by the OED's cross-reference to and definition of hem, or at least: It is entirely possible that it is a vocalization that grew out of vocalizations of lexicalized throat-clearing.

Solution 2:

Being a natural expression with onomatopoeic characteristics its origin is difficult, if not impossible, to trace as suggested by linguist and etymologist Antony Lieberman:

Hmm — or sometimes hm or hmmm — ranks among the words we English speakers say the most, and yet we give it scant thought. It's never defined for us as children, is left out of all but the fattest dictionaries and seems barely a word at all. (It doesn't even have a vowel.) Nonetheless, we all manage to grasp hmm's vast range of connotations.

Although it exists in many languages in a variety of forms, its roots are elusive. "I have no theory of its origin," said Anatoly Liberman, a linguist at the University of Minnesota and an expert on word origins. "Possibly it could have spread from French to English… but you cannot trace it in any way as far as its distant history is concerned, because the word is so natural that it may have arisen at any time."

(www.livescience.com)