Why is there no comma after "no" in "thanks, but no thanks"?
Solution 1:
It would be rare in contemporary usage to insert the (technically correct) comma into "no thanks" or "no thank you".
The words when spoken have no pause after the "no" and it comes out in one lump "nothankyou".
To render it with the comma sounds either stilted or old-fashioned, or there is an emphasis on the "no" which would perhaps indicate that the offer is in some way inappropriate or particularly unwelcome.
That is, a pause after the "no" suggests that the "thank you" needs to be added as an afterthought, because the offer is so egregious that even thanking the person for it is not an obvious response.
In its extreme, it would work something like this:
"I think I have a [medical problem] in my [private body part]."
"Want me to examine it?"
"NO! Thank you."
Solution 2:
The OP is right in assuming that, if the comma were to be inserted, then the sentence would be roughly equivalent to 'Thanks, but no, I refuse the offer; thank you once again'. That is, however, not quite what was intended by the original sentence.
The original sentence has the tone of a relatively casual dismissal of whatever was offered. Such a casual dismissal is incompatible with thanking the offeror very profusely, which would be conveyed by 'Thanks . . . thank you once again'.
The key to understanding the original sentence is to note that thanks by itself can sometimes be used to mean 'no, thanks', if it is accompanied by a suitable gesture, or the context otherwise makes it clear that the offer is being rejected. In 'Thanks, but no thanks', the first thanks is such a thanks of rejection. The speaker then, to remove any ambiguity about the rejection, and perhaps to emphasise it, proceeds to say 'but no thanks'. The way to understand the original sentence is thus as something like 'Thanks, and by that I mean "no, thanks"'.
Now, in 'by that I mean "no, thanks"' we still have the comma. In that rephrasing of the original sentence, the quotation marks surrounding the 'no, thanks' make it easy to see that these two words are to be read together, notwithstanding the comma. However, if we were to remove the quotation marks, as was done in the original sentence, and keep the comma, it would appear that no and thanks perform some separate, independent roles in the sentence; it would be difficult to read them together as one building block of that sentence. It is to avoid that difficulty that the comma is omitted in the original sentence, even though, comma would otherwise be included in 'no, thanks'.