Why were slum kids called “urchins”?
Solution 1:
Looking in the OED, I see two possibly relevant definitions.
1c. A goblin or elf. (From the supposition that they occasionally assumed the form of a hedgehog.)
The first citation for this definition is 1584:
They haue so fraied vs with bull beggers, spirits, witches, vrchens, elues,
There is also
4a. A pert, mischievous, or roguish youngster; a brat.
The first citation of this is 1525:
Come hydyr thou lytyll fole let me see the:
A! it is euen he, by our blyssyd lady!
What lytyll vrchyn hast forgotyn me?
There is also
5a. A little fellow; a boy or youngster; †a child or infant.
But the first two citations the OED gives (1556 and 1600) look like they could easily be classified under definition 4a.
I would guess that 1c, despite the fact that its citations are later, was actually first, and 4a came from that. Calling mischievous kids elves or goblins is an entirely natural etymological progression. And the etymology of 1c is explained by my quote from the OED above.
And going from mischievous kids to street kids is also an entirely natural etymological progression.
Solution 2:
Dictionary definitions of 'urchin', 1699–1790
Urchin appears in slang dictionaries going back to B. E., Gentleman, A New Dictionary of the Terms Ancient and Modern of the Canting Crew (1699):
Urchin, a little sorry Fellow; also a Hedgehog.
Like the entry in B. E.'s slang dictionary, the earliest general dictionary entry that I have been able to find for urchin includes both the insectivore and human senses of the word. From Edward Phillips & John Kersey, The New World of Words: Or, Universal English Dictionary, sixth edition (1706):
Urchin, an Hedg-hog ; also a Dwarf, a little unlucky Boy, or Girl.
The characterization "unlucky" is intriguing but unexplained. Various Kersey dictionaries repeat this definition for the next fifty years, although some—such as Kersey, A New English Dictionary: or, A Compleat Collection of the most Proper and Significant Words, and Terms of Art Commonly used in the Language, second edition (1713) omit the girl from the definition:
An urchin, a Hedge-hog, a Dwarf, a little unlucky Boy.
Nathan Bailey, Dictionarium Britannicum: Or a More Compleat Universal Etymological English Dictionary (1730) reports that urchin comes from a Saxon word (approximately iŋcinᵹ, although I'm not sure that I have reproduced the second character accurately) and reverses the order of the definitions:
URCHIN a little short Boy or Girl ; also a Hedge-Hog.
Samuel Johnson, A Dictionary Of The English Language (1755) identifies a very different etymology for urchin and splits the entry into two distinct definitions:
URCHIN n. s. {heurcuchin, Armorick ; erinaceus, Lat.} 1. A hedge-hog. [First cited example:] Urchins shall, for that vast of night that they may work,/ All exercise on thee. Shakespeare. ... 2. A name of slight anger to a child. [Cited example:] Pleas'd Cupid heard, and check'd his mother's pride:/ And who's blind now, mamma? the urchin cry'd./ 'Tis Cloe's eye, and cheek, and lip, and breast:/ Friend Howard's genius fancy'd all the rest. Prior.
Prior is Matthew Prior, and the poem that Johnson cites is "Venus Mistaken," written circa 1696. The "urchin" in this case is the notoriously mischievous Cupid, whose arrows make him even pricklier than a hedgehog.
Francis Grose, A Classical Dictionary of the Vulgar Tongue (1785) has this:
URCHIN, a child, a little fellow, also a hedge hog.
Interestingly, Grose's A Provincial Glossary, with a Collection of Local Proverbs, and Popular Superstitions, second edition (1790), which focuses on regional colloquialisms rather than slang, has this entry for urchin:
URCHIN. A hedge-hog. North [counties].
It thus appears that by 1790, most of England knew hedgehogs only as hedge-hogs, and reserved the term urchins for children and "little fellows."
'Urchin' as a slang term in 1904
J.S. Farmer & W.E. Henley, Slang and Its Analogues Past and Present, volume 7 (1904) has this useful retrospective discussion of urchin as a slang term:
URCHIN, subs. (old and still colloquial).—1. A mischievous child ; a half-chiding endearment ; 'a little sorry Fellow' (B.E. and GROSE) : also (2) an elf, fairy, or sprite : popularly supposed to take the form of a hedgehog, the original meaning. Hence as adj. =(1) roguish, mischievous ; and (2) trifling, foolish, trumpery.
An in-context instance of 'urchin' from 1556
As Peter Shor's excellent answer notes, instances of urchin applied to a child go back to the first quarter of the sixteenth century. One interesting early instance occurs in John Heywood, The Spider and the Flie (1556):
And with these woordes aside the spider start,/ Where his saide bedfelow and offspring were./ Saying these woordes : now good my owne sweete hart/ And my two babes be ye all of good chere,/ The present cause of all your present fere/ Is past, I haue the caiteffe fast in snare,/ That was the cause of all your fearefull care.
What horeson is it husband, (quoth he) wife/ A fleshe flie as big as a humble bee,/ That shall (if I liue) surelie leese his life./ The yongest spider there, at this cride he,/ Oh, father father I hertelie pray ye/ Remembre when ye shall returne againe,/ To bring me some part of that fleshe flies braine.
How say ye to this babe (quoth the mother)/ Will ye here this vrchin of eyght weekes olde,/ It is a babling brat aboue all other,/ Ye (quoth the father) childe hardely be bolde./ Thornes pricke yonge, that shalbe sharpe folke haue tolde,/ Which sheweth in thee, in that thou art enclinde/ To craue (thus yonge) accordyng to thy kynde.
In this conversation, the spider mother speaks rather disparagingly of her "babling brat" and "vrchin of eyght weekes olde," but it is possible to detect a note of indulgence in her words. The father spider, for his part, seems quite proud of his son's eagerness to obtain a piece of the fly's brain, observing that a thorn destined to be sharp when fully mature will prick at a young age.
'Urchins' as children of the night
Hedgehogs are, of course, nocturnal creatures; and in centuries past, being active at night was generally not viewed as a wholesome way of life. This may help explain the emergence of the supernatural elf- or fairy-like beings termed urchins that Farmer & Henley mentions in its entry for the word. This sense of urchin is quite old (as Peter Shor's answer points out).
Thomas Nashe mentions such urchins in two separate works written in the 1590s. First, in "Strange Newes, of the Intercepting Certaine Letters" (1592), Nashe writes as follows:
I wish unto thee all super-abundant increase of the singular gifts of absurditie, and vainglory : from this time forth for euer, euer, euer, euermore maist thou be canonized as the Nunparreille of impious epistlers, the short shredder out of sandy sentences without lime, as Quintillian tearmed Seneca all lime and no sande ; all matter and no circumstance ; the factor for the Fairies and night Vrchins, in supplanting and setting aside the true children of the English, and suborning inke-horne changelings in their steade, the galimafrier of all stiles in one standish, as imitating euerie one, & hauing no separate forme of writing of thy own ; ...
And second, in "The Terrors of the Night, Or, A Discourse of Apparitions" (1594) Nashe offers this remark:
Fie, fie, was euer poore fellow so farre benighted in an old wiues tale of diuells and vrchins. Out upon it, I am so wearie of it, for it hath caused such a thicke fulsome Serena to descend upon my braine, that now my penne makes blots as broad as a furd stomacher, and my muse inspyres me to put out my candle and goe to bed : ...
The prickly little mammal itself (or a spirit in its guise) appears in Prospero's malediction to Caliban in act 1, scene 2 of The Tempest (1610-1611), cited above in Johnson's dictionary entry:
Prospero. For this, be sure, to-night thou shalt have cramps,/ Side-stitches, that shall pen thy breath up ; urchins/ Shall, for that vast of night that they may work,/ All exercise on thee : thou shalt be pinch'd/ As thick as honey-combs, each pinch more stinging,/ Than bees that made them.
In all of these activities, urchins as intelligent spirits or as dumber animals that ply their unsavory trade at night seem to evoke a certain degree of unease (though perhaps not dread) in the diurnal humans who share stories about them.
Conclusions
None of the early sources I consulted spells out why a child variously described as "sorry," "unlucky," "short," and (above all) "little" should be particularly associated with a hedgehog. Nevertheless, all of those early definitions possess a distinctly critical, commiserating, or patronizing tone. I don't detect in those definitions the note of endearment that often (but not always) accompanies the word poppet during the same period (approximately 1550–1900).
My guess is that urchin as applied to children was never a truly positive or neutral phrase—even before Dickens in Oliver Twist applied it to youthful handkerchief thieves. On the other hand, I don't regard urchin as having a strongly negative sense in many instances. Johnson's characterization of it as "a name of slight anger" comes close to the mark, I suspect, and Farmer & Henley's treatment of it as "a half-chiding endearment" seems apt even for the 1556 instance from The Spider and the Flie.