How did "owly" come to mean irritable or grouchy?
It is actually related to how owls look. They look grumpy! (at least most of them).
There is a strong evidence that it might be originated from Nova Scotia or around that region in colloquial usage:
[South Shore Phrase Book: New, Revised and Expanded Nova Scotia Dictionary By Lewis Poteet (2004)]
[Dictionary of Prince Edward Island English edited by T. K. Pratt (1996)]
The Word Detective also talks about the origin of this term based on a question asked by a Nova Scotian whose mother called her and her sister "owly" when they were in irritable moods. It also relates the term to owl's appearance:
Such is the case with “owly,” which is indeed how it’s spelled. Since at least the mid-19th century, “owly” has been a colloquial term meaning “cranky, cross, angry or fretful.” It’s considered a regional usage, found largely in eastern Canada and the Upper Midwest of the US.
The Oxford English Dictionary lists “owly” first as a synonym of “owlish,” meaning, literally, “resembling an owl,” but usually applied to people who exhibit an unblinking, calm (but often critical) gaze, similar to that of a wide-eyed owl (“The little man with his most owlish air of wisdom,” I. Zangwill, 1895).
“Owly” as a synonym for “cranky” or “irritable” appears to draw on another aspect of the owl’s appearance. Many species of owl have tufts of feathers above their eyes, making the bird resemble a little man with his brow furrowed in disapproval and annoyance. Coupled with the owl’s intense, piercing stare, you have a perfect visual metaphor for someone in a persistently implacable bad mood.
An older dialectal dictionary gives the meaning as stupid and tired for owly; and sleepy and stupid for owlish. The meaning might have expanded in time or this usage might be unrelated and archaic in British English. All the three dictionaries below say that owly is a word in Suffolk dialect.
[English Dialect Dictionary (EDD) compiled by Joseph Wright (1898-1905)]
[A dictionary of archaic and provincial words, obsolete phrases, proverbs, and ancient customs, from the fourteenth century, Volume 2, By James Orchard Halliwell-Phillipps (1852)]
[Dictionary of obsolete and provincial English, Volume 2, By Thomas Wright (1857)]
One of the Urban Dictionary entries asserts that "owly" is common in Nova Scotia: Urban Dictionary s.v. owly (punctuation and capitalization edited)
Out of sorts; grumpy
Used widely in Nova Scotia
When my mother called me to get up and get ready for school, I yelled "Do you have to be so loud?" She said, "Oh, feeling owly this morning, are we?"
ibid, s.v. owlie
To be grumpy, with extreme attitude. Angsty, curmudgeon-like. Think of a grumpy faced owl, that's being owlie.
Man, you're sure owlie this morning... go back to bed!
However, The English Dialect Dictionary: M-Q from 1903 has the following quote:
OWLY, adj. Suf.1 [eu·li.] Stupid; tired.
I 'a bin up all night an fare kienda owly this morning.
The "Suf.1" refers to the following bibliography entry:
Suffolk — Suffolk Words and Phrases. By E. MOOR, 1823.
So that places the word squarely on the left coast of Britain, not North America. However, the meaning is slightly different/more literal: owls, being nocturnal, will blink at you stupidly in the morning, just like a person who has just pulled an all-nighter. The only crankiness involved is implied rather than stated outright: people who haven't slept enough tend to be out-of-sorts and grouchy. (In this sense, I think it is telling that both of the example quotes in Urban Dictionary have to do with getting up in the morning.)
Here's an example that is definitely not literal, and doesn't have anything to do with mornings, but I'm not sure whether it's really using "owly", or just playing on the sound of "scowly". Uncle Wiggily Longears (by Howard Roger Garis, 1915), p.152:
For, off in one corner sat Peetie, and oh! what a scowly-owly look was on Peetie's face. And in another corner was Jackie Bow Wow, and on his face was an even worse scowly-owly look.
The context is that there's no snow for sledding, so the children are in a bad mood/cranky. (Which is precisely the definition offered by Wiktionary, the only other modern dictionary [other than the even iffier Urban Dictionary] where I've found anything on this word.)
The following example seems to show that "owly" has something to do with appearance: The Mascot of Sweet Briar Gulch (by Henry Wallace Phillips, 1908, p.12)
“You’re looking kind of owly, old man—what’s up? Don’t you feel well?”
“Oh, Bud I I’m sick of everything this day—I don’t believe in the constitution of the United States, including the thirteenth amendment, nor the ten commandments, nor the attraction of gravitation, nor anything else—it’s all a damned lie.”
However, other examples don't seem to be based on appearance at all: American Mercury Magazine 1945, p.58 (snippet view only, so I can't tell who the author is)
"I was owly," he confessed. "I got so goddam low in the mouth and owly that I lost my head and wrote it. As soon as I mailed it, I knew it was a mistake."
Many of the examples I've found for this adjective are literal, in the sense that they're calling someone owl-like, whatever that means. Some of the examples are clearly derived from an owl's stereotypical daytime sleepiness, and as any night-owl living in this morning-person world will tell you, sleepiness = crankiness. The "feeling owly" examples seem to support this derivation. However, the "scowly-owly" and "looking kind of owly" examples imply a derivation based on an owl's appearance. I think the true answer is a bit of both: owly means grumpy both because owls often look grumpy, and because owls don't like to get up in the morning and thus are grumpy.