Origin of "kettle of fish"
Solution 1:
The phrase means, as you said, 'a different thing.' According to this website:
There was, it seems, a custom by which the gentry on the Scottish border with England would hold a picnic by a river. The custom was described by Thomas Newte in his Tour of England and Scotland in 1785: “It is customary for the gentlemen who live near the Tweed to entertain their neighbours and friends with a Fete Champetre, which they call giving ‘a kettle of fish’. Tents or marquees are pitched near the flowery banks of the river ... a fire is kindled, and live salmon thrown into boiling kettles”.
The way the phrase became an idiom, however, is not clear. Visit that site to read more about possible explanations of the idiom.
Solution 2:
The British idiom a different kettle of fish and a whole new kettle of fish is related to the North American idiom a whole new ball game. The latter means “a situation that is completely different from a previous one”, whilst the former means “to be completely different from something or someone else that has been talked about”.
Nowadays the term kettle is usually associated with teakettles, but in the past they were many different kettles for boiling or cooking things, so a kettle was basically any cooking pot or pan.
The website Online Etymology Dictionary says that kettle is probably derived from Latin catillus a "deep pan or dish for cooking". The inherited English form would have been *chettle due to palatalization, but the initial consonant was changed back to a k- (attested from around 1300), probably under the influence of its Old Norse cognate ketil.
One of the earliest instances of fish-kettle recorded by Google Books is dated 1747 in The Art of Cookery, Made Plain and Easy; Which Far Exceeds Any Thing of the Kind Ever Yet Published. By Hannah Glasse
To boil a Cod's Head.
Set a Fish-Kettle on the Fire, with Water enough to boil it, a good Handful of Salt, a Pint of Vinegar, a Bundle of sweet Herbs, and a piece of Horse-raddish; let it boil a Quarter of an Hour, then put in the Head, and when you are sure it is enough, lift up the Fish-Plate with the Fish on it, set it across the Kettle to drain, then lay it in your Dish and lay the Liver on one side. Garnish with Lemon and Horse-raddish scraped; melt some butter, with a little of the Fish Liquor, and Anchovy, Oysters, or Shrimps, or just what you fancy.
The Phrase Finder says the older expressions a fine kettle of fish and a pretty kettle of fish derive from the noun kettle of fish whose first cited reference is dated 1785 in Thomas Newte's A Tour in England and Scotland. However, ...
'A different kettle of fish' is much later in origin than 'a pretty kettle of fish' and is known only since the 1920s. It's quite pleasing that, as far as etymology goes, 'a different kettle of fish' is a different kettle of fish.
The London Mercury, Volume 8 (1923)
Solution 3:
A fine or pretty kettle of fish
As Peter Shor's comment beneath Ralph Richardson's answer indicates, "kettle of fish" has been used as a slang term for several centuries. The same definition of the term that he points to appears in Francis Grose, A Classical Dictionary of the Vulgar Tongue, second edition (1788):
KETTLE OF FISH. When a person has perplexed his affairs in general, or any particular business, he is said to have made a fine kettle of fish of it.
The earliest instance of the expression that a Google Books search finds is from "The Rival Masons: A Dialogue Between Mr. D———, Mr. H———, and a citizen of London," in The Political State of Great Britain (December 1738):
H. ... I doubt [that is, fear] we have but rouz'd a sleeping Lion : A stop-Thief has sometimes saved a House-breaker ; and many a Wench has sav'd her Reputation by crying Whore first : But the more this Matter is stirred, the more it stinks, and I doubt we have made a fine Kettle of Fish on't.
This pretend dialogue is unusual for the multitude of proverbs and idiomatic phrases that the two knavish masons pitch back and forth in this part of the dialogue—"a Fool's bolt is soon shot," "When Knaves fall out, Honest Men come by their Rights," "a Word to the Wise," "Penny wise and Pound foolish," "putting a Spoke in my Wheel"—but as testimony taken at a trial for adultery adjudicated on December 5, 1738, suggests, "a fine kettle of fish" may not yet have been broadly familiar to English people. From The Tryal of a Cause for Criminal Conversation: Between Theophilus Cibber, Gent. Plaintiff, and William Sloper, Esq; Defendant (1739):
Mrs Hayes. ... One Day after I gave warning [that the plaintiff's wife's maid must vacate the room she had rented for her mistress to use for assignations with the defendant], Mr Sloper was in a great Passion above Stairs at something, and Mrs Hopson [the maid] came to me, You have made a fine Kettle of Fish of it, says she. I did not know what she meant by her Kettle of Fish. What Fish do you mean? says I. Why there, says she, you have been talking of Matters, and he's stark mad at it above Stairs.
The expression also appears (a decade later) in Henry Fielding, The History of Tom Jones a Foundling (1749), in the course of a tirade by Mr. Western to Mr. Allworthy:
'There you have done a fine Piece of Work truly. You have brought up your Bastard [Tom Jones] to a fine Purpose ; not that I believe you have had any Hand in it neither, that is, as a Man may say, designedly ; but there is a fine Kettle of Fish made on't up at our House.'
In all of these instances, muddle seems to be a fair translation of the intended sense of "fine kettle of fish." In explaining the origin of the phrase, J.C.M.H., "Slang Terms and Their Derivations," in Bailey's Monthly Magazine of Sports and Pastimes (February 1872) argues that the kettle in question is not cookware:
We talk of a 'pretty kettle of fish,' and kettle is vulgarly supposed to refer to the culinary vessel in which fish is boiled ; this, however, is a mistake, as kettle is a kind of net in which fish is caught, and 'a pretty kettle of fish' merely means a bad catch.
Ebenezer Brewer, Errors of Speech and of Spelling, volume 1 (1877), endorses the same explanation, arguing that kettle is a corruption of kiddle ("a basket for catching fish"); in this, Brewer echoes the earlier suggestion of Anne Baker, Glossary of Northamptonshire (1854). A comment in Gentleman's Magazine (March 1823) begins its discussion of "kettle of fish" with the description of a Fête-champetre, as described in Newte's Tour (1785), cited in Mark's answer above, but then speculates in a different direction:
We much question, however, that it is an explanation at all, and for this reason: Fête-champetre would be doubtless a jovial affair;—how, then, does it happen that "kettle of fish" is a phrase never used but when something is spoilt or done amiss, thus: "you have made a pretty kettle of fish of it!" It is a question for the commentator on proverbial expressions, whether this application of "kettle of fish" to blunders or mismanagement arose from some miscooking of salmon at any of the Fêtes-champetres alluded to. The common expression "you have made a mess of it" evidently originates in some unfortunate overboiling.
A different kettle of fish
Since "another kettle of fish" and "a different kettle of fish" emerged as idioms more than a century after "a fine/pretty kettle of fish" did, the meaning of the new phrases was probably at least tinged with the notion that the metaphorical kettle of fish in question might be problematic. That is, "another kettle of fish" didn't simply mean "another thing altogether" but "another thing—and one that may be troublesome to put in order." This sense of the phrase is supported by its first Google Books appearance, in 1860. From Oliver Optic, Marrying a Beggar (1860):
"I had almost forgot to mention that brother Joseph had arrived in New York, and telegraphs that he shall be here to-night by the New Haven train."
"Just like you! Never tell of a thing till the last moment!" said the lady petulantly.
"I received the despatch only two hours ago."
"Here is another kettle of fish," continued the lady, musing. "That everlasting niece of yours is in the way again."
But very quickly the "another thing altogether" meaning seems to have taken over. From Harry Hooper, "Left on Board," in Under the Crown (February 1869):
"A southerly wind and a cloudy sky" may be a very pleasant theme for fox-hunting squires in dear Old England, but when a man is under a cloud in a foreign country, with a southerly wind in his pockets, and Mary Thompson's mark, " M. T." on his clothes chest, then it's quite another kettle of fish. This, however, was my sad case.
From "Parliamentary Jottings," in Touchstone: A Saturday Journal of Criticism, Commentary, and Satire (October 30, 1869):
The faithful minister, we are told, may always rely on adequate and generous support, and if at any time, ... he should be in a state of semi-starvation, he ought rather to like it than otherwise, especially if he has a wife and children. We must pay the policemen, the Judge and the gaoler, but those who do more than any other class of men in the community to diminish the need for either, must be deprived of the scanty pittance doled out to them of late years with such bad grace. But the legislator is altogether another kettle of fish. He is such an etherial being, that we must not force him to gain a living by contact with the harsh, coarse world.
And similarly with "a different kettle of fish." From Francesco Abati, See-Saw, volume 2 (1865):
"Well, of course you had a great advantage over me so far. ...There's nothing done here in a regular way, except a little gout and rheumatism. Accidents, of course, one can never depend upon, and what's the use of a case like this once in five years? There couldn't be a worse county than this. Now, when I was a young man practising in Smithfield, and taking in free patients, it was a very different kettle of fish. My goodness!” said the doctor, voluptuously smacking his lips, “ that was a neighbourhood. Something like disease there—and a nice variety of it, too! ..."
And from an 1868 translation of Goethe's Egmont:
Jetter. ... Have you heard one of these preachers?
Soest. First-rate fellows! A short time ago I heard one of them preaching in a field before thousands and thousands of people. A very different kettle of fish, I can tell you, to that which our pulpit droners give us, smothering the people with their scraps of Latin. My man spoke out as if he meant it.
Conclusions
To judge from Google Books results, "a fine/pretty kettle of fish" (meaning "a muddle") has been idiomatic in English since at least 1738, and "a different/another kettle of fish" (very quickly meaning simply "quite another thing") has been used since at least 1860. The latter expressions seem very likely to have emerge from the former ones, the shift in meaning appears to have be swift and substantial.
Solution 4:
I have the earliest attestation for the idiom dated to 1742 in Oxford Dictionary of English Proverbs. The work is Henry Fielding's novel The History of the Adventures of Joseph Andrews and of his Friend Mr. Abraham Adams.
With such like discourses they consumed near half-an-hour, whilst Betty provided a shirt from the hostler, who was one of her sweethearts, and put it on poor Joseph. The surgeon had likewise at last visited him, and washed and drest his wounds, and was now come to acquaint Mr Tow-wouse that his guest was in such extreme danger of his life, that he scarce saw any hopes of his recovery. "Here's a pretty kettle of fish," cries Mrs Tow-wouse, "you have brought upon us! We are like to have a funeral at our own expense." Tow-wouse (who, notwithstanding his charity, would have given his vote as freely as ever he did at an election, that any other house in the kingdom should have quiet possession of his guest) answered, "My dear, I am not to blame; he was brought hither by the stage-coach, and Betty had put him to bed before I was stirring."--"I'll Betty her," says she.--At which, with half her garments on, the other half under her arm, she sallied out in quest of the unfortunate Betty, whilst Tow-wouse and the surgeon went to pay a visit to poor Joseph, and inquire into the circumstances of this melancholy affair.
Project Gutenberg