Meaning and origin of "That dog don't hunt"

According to the Phrase Finder:

It's modelled on the 17th-18th century phrase "that cock won't fight". In the days of cock-fighting, a cock that wouldn't fight when out into the pit was a natural metaphor for a plan or theory that simply wouldn't work. A similar sporting metaphor from horse-racing is used when we say that a plan or theory "isn't a runner".

source

You use the phrase to describe an idea which won't work. It's a prediction of failure.

Example:

Person A: I heard you are going to ask Jane out on Friday.
Person B: I was going to ask her out, but after I found out she's seeing someone, I decided not to. That dog won't hunt.

Meaning

That dog won't hunt is a US phrase meaning something won't fulfil its intended purpose, or a plan or scheme will fail.

Origin

It is southern US slang, originally that old dog won't hunt meaning something just isn't going to happen.

The Facts on File Dictionary of Clichés by Christine Ammer says:

that dog won't hunt

This idea or excuse won't work.This folksy expression originated in the American South, where dogs are commonly used to hunt raccoons and other wild animals. Also put as that old dog won't hunt, it originated in the late 1800s.

The Facts On File Dictionary of American Regionalisms by Robert Hendrickson says:

that dog won't hunt Common in the Ozarks and elsewhere for anything, especially a plan or idea, that won't work, that isn't practical. "It looks good on paper, but that dog won't hunt."

Examples

This longer version appears in a summary of American Speech: Volume 14 from 1939:

'If the' ain't no fools, the' ain't no fun,' said usually in self-derision; and 'That old dog won't hunt,' meaning that an excuse offered will not serve. These and the numerous specimens which follow have simply been grouped by the present writer under the heading of Miscellaneous, explanations being made only when the meaning is not clearly evident.

A good example of an ever longer version appears in the 1961 The Gay Place: Being Three Related Novels by Billy Lee Brammer, from Texas, quoting a governor:

"Knew enough to tape-record that conversation. That's knowin' somethin'. That's knowin' more'n I know . . . You even made him count out that goddam money right out loud so it would get on the tape. Why hell, man! You know. Don't give me that stuff. That old dog don't hunt no more — that cow's been bred and milked and damn near slaughtered."

Brammer began writing the novel whilst working for Texas senator Lyndon Baines Johnson.

A summary of a 1965 Automobile Cases says:

In the homely idiom of a colorful West Kentucky lawyer long gone but not forgotten, "that dog won't hunt." To begin with, the word "insured" as it was held to apply to Latham under the omnibus clause of American's policy expressly includes not only the driver but also "any person or organization legally responsible for the use thereof," ...

It also appears in Jesse Hill Ford's 1965 novel The Liberation of Lord Byron Jones, set in Tennessee:

"Calling her house help Mr. and Mrs. just won't get it! That old dog won't hunt," Oman had confided. "I know, but let her discover it herself," Steve had said.

That dog won't hunt was further popularised in US political rhetoric of the 1970s, especially by Texan and former president Lyndon B. Johnson.

It appears in an article published in both the Sarasota Herald-Tribune and The News and Courier of February 7, 1970. In a televised interview with Walter Cronkite, Johnson said:

On the Goldberg proposal to stop all the bombing, Bunker "came back strong and said, 'I just can't. That dog won't hunt. We just can not get that over, it would just blow everything .'