Origin of "dog(ging) it"?

J.E. Lighter, Random House Historical Dictionary of American Slang (1994) confirms the gist of the American Heritage Dictionary's coverage of the phrase "dog it":

dog it, 1. a. Esp. Sports. to hang back, do less than what is required or expected; (specif.) to loaf, shirk, or renege. [Citations from 1905 forward omitted.] b. to proceed slowly. [Citations from 1925 and 1970 omitted.] 2. a. to leave quickly or furtively; run away. [Citations from 1931 forward omitted.] b. ... to go on foot; walk. [Citation from 1973 omitted.] 3. to put on the dog, ...; show off. [Citations from 1932 and 1938 omitted.]

Early more-or-less literal meanings of 'dog it'

The earliest instance of "dogging it" that an Elephind newspaper database search finds carries the literal sense "set a dog on it," as we learn an account of monetary losses that the writer blames on poor fencing between a farm owned by a man who raised cows, pigs, and other animals, and the writer's corn farm, in "Fences," in the [Ohio County, Indiana] Rising Sun (June 4, 1836):

To price of a hog of my neighbor Hodge for which I had to pay, having dogged it in my cornfield, so that it died, $3.

To time lost in attending a law suit, about said hog, and costs of suit, $5.

To loss of a valuable Dog which I supposed Hodge had killed, in revenge for the killing of his hog by said dog, but which I could not prove, $5.

Another early meaning of "dog it" appears in Delia Bacon, "The Elizabethan Men of Letters" the Sydney [New South Wales] Morning Herald (September 17, 1857), where the clear sense is "behaving like a [hostile] dog":

Was it strange that these men should find themselves without sympathy in an age like that?—an age in which the masses were still unlettered, callous with wrongs, manacled with blind traditions, or swaying hither and thither, with the breath of a common prejudice or passion, or swayed hither and thither by the changeful humours and passions, or the conflicting dogmas and conceits of their rulers. That is the reason why the development of that age comes to us as a literature. That is why it is on the surface of it Elizabethan. That is the reason why the leadership of the modern ages, when it was already here in the persons of its chief interpreters and prophets, could get as yet no recognition of its right to teach and rule—could get as yet nothing but paper to print itself on, nothing but a pen to hew its way with, nor that, without death or danger dogging it at the heels, and threatening it, at every turn, so that it could only wave, in mute gesticulation, its signals to the future.

This usage—in the longer form "dogging it at its heels"—is indistinguishable in effect from the modern idiom "snapping at its heels." Although this form of the expression may share in common with more-recent senses of "dogging it" the core allusion to "behaving like a dog," the particular canine behavior involved is evidently quite different. Moreover, the "it" attached to "dogging" has a particular object in mind, whereas in the later senses the "it" is nebulous.

A narrower sense of "dogging it"—"following [something] the way a dog would"—appears in "The Great Nebula in Orion," in the Alexandria [Virginia] Gazette (February 5, 1883):

Professor Young says the companion [celestial object] is rapidly approaching Sirius. But what is it? We can hardly call it a sun, for it has not the light of a sun. It may be the ember of a sun that is dying out It may be one of those strange dark bodies with which space, it is believed, abounds, and which having fallen within the attraction of Sirius is dogging it through the sky.

Here, of course, the temptation to use "dogging it" is probably not unrelated to Sirius's identity as the Dog Star. Again the "it" is something very definite.

Early figurative meanings of 'dog it'

An intriguing figurative instance of "dog it" appears in an untitled item in the [Lincoln, Nebraska] Capital City Courier (December 19, 1891, where the full expression is "yellow dogging it":

If the judgment of those Lincolnites who saw the Aronson Opera company in "Uncle Celestir" in Omaha can be relied upon, there was little loss in the cancellation of the date for this city. Thu piece is not an opera but a musical comedy and the management is now yellow dogging it through the country prior to presentation in New York.

It's hard to say exactly what "yellow dogging it" means as used here. I am inclined to read it as meaning something like "dragging it [the musical comedy] like a sorry yellow dog"—but that may not be accurate. The term "yellow dog" has a long record as a pejorative in U.S. English. J.S. Farmer & W.E. Henley, Slang and Its Analogues past and Present (1904) has this short but vague entry for it:

Yellow-dog, subs. phr. (American). — A strong term of contempt.

And Mitford Mathews, A Dictionary of American English on Historical Principles (1951) has this:

yellow dog. Chiefly in slang uses. 1. A cur or mongrel usu. in transf. uses as a symbol of utter worthlessness. Also attrib. [Citations from 1835 forward omitted.] b. A contemptuous designation for a person. [Citations from 1880 forward omitted.] ... 3. yellow dog under the wagon, and variants (see quots. ...) [First cited quotation:] 1857 Spirit of the Times 19 Dec. 248/1 For Potomac's pedigree see page 407 of 'Edgar's General Stud Book,' which is about as long and reliable as that of 'the big yellow dog under the wagon.'

The sense of "big yellow dog under the wagon" seems to be "a thing of which very little is known, but about which it is not safe to make assumptions." Whatever the sense of "yellow dogging it may have been, the 1891 Capital City Courier instance of that expression is the only one that my Elephind searches uncovered.

Another intriguing but ambiguous instance occurs in "Dramatic Notes" in the [Washington D.C.] Morning Times (October 4, 1896):

Although Miss Georgia Cayvan makes her metropolitan debut as a star in New York tomorrow night, she has been "dogging it" all week. She first acted on any stage in Haverhill, Mass., and in the same town she first appeared as a star.

There is no further context offered for the use of "dogging it" in this item.

[[UPDATE (July 20, 2018): Spurred by a comment by EL&U participant (and poster of this question) Colin, I checked to see whether dog might not have had a particular meaning in late-nineteenth-century show-biz argot—and evidently there was. From Don Wilmeth, The Language of American Popular Entertainment: A Glossary of Argot, Slang, and Terminology (1981) [combined snippets]:

Dog: 1. Circus slang for a hippopotamus. 2. Obsolete theatrical term for an audience outside New York City. To try out a play on the road was once to try it out on the dog. see also TRY IT (OUT) ON THE DOG.

...

Try it (out) on the dog: To test a theatrical piece, usually outside of the city where it would eventually be given its principal production.

The two newspaper examples reported in this section of my answer reflect exactly those old show-business senses of "dogging it." Thanks, Colin, for the excellent observation.]]

Early instances of 'dog it' in horse racing and other sports

The first sports reference to "dogging it" appears in "Ninety-Eight for Old Logan," in the San Francisco [California] Call (April 2, 1897), where the meaning seems to be "losing spirit," "accepting defeat," or "refusing to go all out":

With his well-known preference for muddy going, Barney Schreiber's Sweet William was backed to beat the favorite, Caliente, in the fourth race over seven furlongs, decided under selling conditions. William could not shake Stemler's colt off, and "dogging" it through the stretch was even defeated for the place by Russella. Caliente won easily by four lengths.

The "verb form "dogged it" appears less than six months earlier in the same newspaper. From "Joe Terry Likes Muddy Going Now," in the San Francisco [California] Call (November 21, 1896):

There were but three starters in the seven-furlong handicap, Moylan carrying 113, Sir Play 110 up and Grady in at 90 pounds. Moylan ruled favorite at 13 to 20, and, galloping away in front at the start, looked an easy winner, but "dogged" it badly the last sixteenth, and was beaten out cleverly a length by Sir Play, which horse was splendidly rated by Willie Martin.

Poor Moylan is accused of dogging it again ten weeks later in "Pat Murphy Was on His Mettle" in the San Francisco [California] Call (February 6, 1897):

The cause of all the gibberish was the nose-to-nose termination of the fourth race, in which Pat Murphy gained the decision over Moylan. Some were positive that Moylan won by a head, while others made it a length. The fact of the matter was that opposite the paddock Moylan looked a handy winner, but "dogged" it badly the last fifty yards, and the fast-coming Pat nipped him a nose on the wire.

The San Francisco Call contains nine additional instances of horses who "dogged it" within the next two years—on November 17, 1897, on December 19, 1897, on March 31, 1898, on April 2, 1898, on November 18 1898, on December, 2, 1898, on December 7, 1898, on January 24, 1899, and on January 28, 1899. Thereafter, other newspapers—such as the Kansas City [Missouri] Journal and the Omaha [Nebraska] Daily Bee—both carrying the same report on February 26, 1899, from horse races in New Orleans, Louisiana, began to include the term.

The first Elephind match in which a news story applies "dogging it," in the same sense as with horses, to a person is in "Washington Fared Well in Inter-City Boxing," in the Washington [D.C.] Times (February 17, 1906):

Tipman has a weak jaw and Lowe played for that. Every time he landed a wallop on the head the Baltimorean backed up, and if Lowe had not noticeably slowed up in his work in the last four rounds he would undoubtedly have put out the pride of Rock street. As a matter of fact, Tipman began dogging it after that cruel fifth round, and should have been an easy mark at any time thereafter.

Conclusions

Figurative use of "dogging it" appears in newspapers beginning in 1891, but the sense of the phrase, as used in two early nonsporting contexts, does not appear to be related to the sporting sense. Instances of "dogging it" and "dogged it" in the now familiar sense of "not trying hard" or "didn't try hard" begin to appear in U.S. newspapers in 1897—and they are highly localized early on.

Elephind searches turn up a dozen unique instances of "dogging it" or "dogged it" between November 21, 1896, and January 28, 1899—all in the same newspaper (the San Francisco Call) and all in the context of racehorses that turned in poor performances relative to expectations or after starting out well—before it appears in a similar sporting context anywhere else.

It seems highly likely that the popularization of this particular slang or idiomatic expression originated with its repeated use in to horse-happy Call.