Etymology of the expression "to entertain an idea"
In earlier senses, the word entertain can refer to supporting or maintaining something (Oxford English Dictionary, "entertain, v.," I), to holding things together (II), or to receiving or admitting something (III). Many kinds of things can be received or admitted: hosts entertain guests, for instance. Ideas and requests can also be entertained, as def. 9b (grouped under III) suggests:
b. To give consideration to (an idea, request, etc.); to think about, contemplate.
1583 B. Melbancke Philotimus (new ed.) 201 I straightlie inhibit you for your better thriuing, neuer once to enterteine one thoughte of Aurelia.
1605 Bp. J. Hall Medit. & Vowes II. §53 But I will suspect a nouell opinion of vntrueth; and not entertaine it, vnlesse [etc.].
1665 R. Boyle Occas. Refl. ii. xi. sig. P7v Who thinks it not time to entertain thoughts of Death.
Ideas, thoughts, and opinions could all be entertained, that is, considered or contemplated. For instance, in The Secretaries Studie (Thomas Gainsford, 1616; link to 1652 edition), where the speaker describes their own constancy against entertaining a thought of changing their mind:
for why should you giue me any cause of suspition, when a warrant is sealed of our happinesse; or assume to your selfe a counterfeit libertie of trying my patience, when you know i will pul out mine owne heart, before it shall entertain a thought of alteration
The way we most often use entertain now, meaning to engage or amuse (IV), is related to these other three meanings, but they all emerge as distinct meanings fairly early (in the sixteenth century). Taking its literal Middle French meaning entretenir (entre- ~ among, -tenir ~ hold), each modern meaning is only a couple of steps away from holding something. So it's likely that III and IV distinguish themselves at this early moment in time by the kind of object they take; entertaining someone came to refer to holding them in a state of amusement or engagement. Entertaining an abstraction came to refer to holding it in one's thought.