What is the origin of "A cat in hell's chance"
Solution 1:
I found 13 instances of "cat in hell" in Google Books from 1759-1820. All but two refer to said cat being "without claws".
I suspect it was a long-standing spoken idiom before turning up in print - the idea being that Hell is assumed to be an exceptionally hostile environment. Although cats are remarkably well-adapted for survival (nine lives and all that), their claws are formidable weapons in that context.
In an earlier age where cats were largely expected to feed themselves by catching mice, etc., a cat would have a hard time surviving anywhere without its claws. It certainly wouldn't be expected to thrive in Hell.
Over the past century, the alternative not a snowflake/snowball in hell's chance has become increasingly popular. The snow variants, collectively more common now than the metaphoric cat, obviously allude to the standard perception that hell is exceptionally hot.
Solution 2:
FumbleFingers explains the meaning well, here's some extra notes. I also noticed a fair number of citations from the navy.
18th century
The original phrase was "no more chance than a cat in hell without claws" (with variation) and (taking a different tack and searching for "hell without claws" rather than "cat in hell" turns up some with a few mis-scanned "cat") I can find no earlier occurrence than the 1759 The life and real adventures of Hamilton Murray, written by himself linked in FumbleFingers' answer:
From the same search, we can find this phrase in Francis Grose's 1788 A Classical Dictionary of the Vulgar Tongue:
A humorous poem in The Gentleman's Magazine, for April, 1792, written from the point of view of a cat in hell, sent to his master on earth, has a footnote that claims a (tongue-in-cheek?) origin:
The first that doesn't mention claws is 1783's A Letter to Sir Phil. Jen. Clerke
19th century
In the 19th century, John Bellenden Ker "endeavoured to retrace the original form of the words which I believe to have then duly conveyed the sense of the phrases of the above category. By applying the sound of the words which constitute the modern phrase to others which it fitted in the Low Saxon stage of our language, I have always found a sense, corresponding with that conveyed by the form under which they are now disguised, to be the result of the experiment. The following pages contain the proofs of this test."
Well, I'm not entirely convinced by the "proof", but here it is from 1834's An essay on the archaeology of our popular phrases, and nursery rhymes:
And by 1837 Bellenden Ker produced another edition with a whole page of explanation, which I won't reproduce here, but his digging resulted in this "origin": so that the existance of villainy contains within itself a clear proof there must be a hell.
The 1892 Journal of American Folklore reports the phrase used to mean lack of peace rather than lack of chance:
There 's no more peace here than for a cat in hell without claws. (Ohio.)
However, still the most common meaning was to have no chance, to be helpless or defenceless.