How did “pumpkin” come to be a term of endearment?

The logic of some terms of endearment is fairly clear. Sweetie, honey, cupcake all refer to food treats. However, the use of the term pumpkin as a tenderness seems somewhat counterintuitive. While reasonably tasty and the basis for making some treats (pumpkin bread, pumpkin pie), on its own it is a rather prosaic vegetable. Its appearance is also not obviously in the category of beautiful flora.

Dictionary references are not much help. Only two of the online dictionaries in onelook.com list the affectionate definition. Collins offers, without etymology or example,

often capital (mainly US) a term of endearment

Wiktionary gives one definition as

(US) A term of endearment for someone small and cute.

It cites only to the lyrics of a song by John Prine from 1991, Daddy’s Little Pumpkin.

A search on Google for the term little pumpkin (a search for pumpkin is beyond my patience) shows a fair amount of usage in the later 20th century and the 21st century as an affectionate nickname. One of the earliest, in that period, appears in the 1951 novel by Myron Brinig, The Sadness in Lexington Avenue

Let me hear it. Frieda, my little pumpkin, my little sugarplum

There are few 19th century references using pumpkin, somewhat sweetly, but not quite as an endearment.

In 1867, The Little Corporal, a children’s magazine published a story that included

for Matie was almost as round as a little pumpkin

This appears to be a straightforward description, not very complimentary, about shape.

In Forrest’s Illustrated Juvenile Keepsake,1851 there is reference to another apparently roundish child

If a little pumpkin, like Dumpy Dorcas, had rolled upon the snow, what harm would have come of it?

In The Child’s Friend and Family Magazine from 1858, there is a reference to an American Indian character in a story whose English name appears to be Little Pumpkin.

So, whence the acceptance (by some) of pumpkin as an affectionate reference for the apple of your eye?

Terms of endearment reflect what the user finds desirable, and not everybody shares the (American?) view that sweets are the most pleasant food. The French mon petit chou means literally 'my little cabbage', Russians (sometimes) call each other 'my little wild raspberry', and several pages could be written (by someone with a strong stomach) on German food endearments; e.g. mein Spätzle, the name of a type of pasta, apparently derived from 'wild sparrow'. The prize as far as I know, however, goes to a respondent to a BBC enquiry for endearments:

I am married to a Tibetan lady who calls me nyingdu-la. This translates as "most honoured poison of my heart". Adam Buckley, Hebden Bridge, UK

The OED has pumpkin as a term of endearment from 1900 in Dialect Notes:

Pumpkin, a student's best girl.

This is their sense 2c. Sense 2 includes other chiefly North American colloquial and figurative uses, such as 2a) "Applied contemptuously to a person who is stupid, conceited, or self-important, or (occas.) to a stout or portly body" (1680) and 2c) "In predicative use: a person or matter of importance or consequence; an impressive thing. Esp. in some pumpkins" (1845).