Origin of "bite your tongue" as a response
Here's but my position was delicate, and I bit my tongue and was silent from 1893, so it's obviously been around a while (but I held my tongue has always been far more common).
And here's he stopped and bit back his anger from 1945, showing how "bite" has long been used metaphorically in the sense of "restrain" (what's being bitten needn't in fact be the tongue).
But when people respond to a cutting remark with "Bite your tongue!" I would say they're simply introducing a creative variation on an established idiom. The sense there is "You should punish your tongue for saying such a thing!".
It's somewhat similar to "Wash your mouth out! {with soap and water}", used to mean something like "Your mouth must be unclean to have said such a thing!"
The notion of biting one's tongue instead of speaking or taking action goes back (in English) at least to Shakespeare. From Henry VI Part 2 (1591):
Pirates may make cheap penn'worths of their pillage,/ And purchase friends, and give to curtezans,/ Still reveling, like lords, till all be gone:/ While as the silly owner of the goods? Weeps over them, and wrings his hapless hands,/ And shakes his head, and trembling stands aloof,/ While all is shar'd, and all is borne away;/ Ready to starve, and dares not touch his own./ So York must sit, and fret, and bite his tongue,/ While his own lands are bargain'd for, and sold.
Biting one's tongue to stifle the urge to speak also seems to have a long history in Spanish. Cervantes uses it several times in Don Quixote and in his [A Dialogue Between Scipio and Bergansa, Two Dogs belonging to the City of Toledo*. From a 1719 translation of Cervantes's Don Quixote:
I beg your Pardon, answer'd Montefinos, Signior Don Quixote, I might have guess'd indeed that you were the Lady Dulcinea's Knight, and therefore I ought to have bit my Tongue off, sooner than to have compared her to any thing lower than Heaven it self.
And from a 1767 translation of A Dialogue Between Scipio and Bergansa:
Bergansa: ...What I said was not to lay a law upon me, but only a bare promise, that I would bite my tongue, whenever I censured anything; and now-a-days such things are not so strictly observed as formerly; for today a law is made, and tomorrow it is broken, because perhaps it suits not with our conveniency to keep it; and in on moment we promise to amend our faults, and the next fall into greater; it is one thing to commend good laws and regulations, and another to submit ourselves to them; in a word, saying and doing are two things; let the devil bite himself if he will, for me, for I am not such a fool as to bite my own self, nor practice things upon a mat, where there is no one sees me, who may applaud my heroic actions.
Scipio: I find by this, Bergansa, that if you were a man, you would be a hypocrite, and that all your deeds would be only in outward appearance, done in the eyes of the world, and covered with the cloak of virtue, that you might gain the praise of good men, as in general all hypocrites endeavor to do.
Bergansa: I know not what I should do then, I am only sensible what I shall do now, which is, that I will not bite my tongue, having so many things to say, that I know not how nor when I shall be able to finish them; and more so, seeing I am afraid, left at the break of day our speech should be taken from us again, and then we must remain in the dark, as to all these things.
A similar notion appears in Jean-Jacques Rousseau's Eloisa. Here is a sentence from a 1761 translation of that book:
My mouth was open to add that the castle was like that of Lord B_____, —who ... but luckily I had time to bite my tongue.
I suspect that the original notion of biting one's tongue was more vivid and violent than the mere idea of silencing oneself by inflicting a sharp nip on a delicate body part. This example, from Susanna Moodie, Geoffrey Moncton: or, The Faithless Guardian (1855) suggests the possibility of rendering oneself incapable of speaking by actually biting off one's tongue:
I could have bitten my tongue off for my want of tact, but the blunder was out, and she answered with some asperity.
The character in Moodie's book is obviously exaggerating, but other sources take the threat literally. Thus Wellins Calcott, A Candid Disquisition of the Principles and Practices of the Most Ancient and Honorable Society of Free and Accepted Masons (1769) cites an example from Pliny:
Annaxarchus, who (according to Pliny) was apprehended in order to extort his secrets from him, bit his tongue in the midst, and afterward spit it in the tyrant's face, rather chusing to lose that organ, than to discover those things which he had promised to conceal.
And from a 1784 collection of old ballads, we have "The Spanish Tragedy; containing the lamentable murder of Horatio and Bellimperia: with the pitiful death of old Hieronimo {Part 2}":
To torture me they did prepare,/ Unless I should it straight declare./ But that I would not tell it then,/ Even with my teeth I bit my tongue,/ And in despite did give it them,/ That me with torments sought to wrong.
As for the question of when someone first invited someone else to "Bite your tongue," here is the last quatrain of a poem anachronistically titled "The Telegraph Inverted" from John Lauderdale, A Collection of Poems, Chiefly in the Scottish Dialect (1796):
Not meaning further for to teaaze ye,/ I on my hono'r here advise ye,/ 'Afore ye tak' a side that's wrong,/ To just sit down and bite your tongue.
The "telegraph" of this poem appears to be something akin to a telescope, since the poet says that he "rais'd it up to tak' a view,/ Just i' the end ye looket thro'."
UPDATE (8/26/14):
In a follow-up Google Books search on this subject, I came across this detailed discussion from Thomas Wilson, John Bagwell, and Andrew Simson, A Complete Christian Dictionary (1678):
To gnaw their tongues] To take most grievously the fall of their pomp, dignity, and authority ; also furiously, for extreme sorrow, to bite their own tongues. Rev. 16. 10. And they gnawed their tongues for sorrow. Some do expound this, of renouncing their own proud words and writings, But the former signification is better.
To be in great agony, like those that bite their tongues for grief and anger, ready at the next increase of pain to devour or destroy themselves. Or, it may be meant, They shall eat their words and be ashamed of their railings upon Gods Saints, though they leave not their Idolatry, vers. II. Annot.
This description suggests that 87 years earlier Shakespeare did not intend, in using “bite his tongue” in Henry VI Part 2, to suggest that York would have to bite his tongue to avoid crying out or saying something inappropriate or unwelcome, but that he would have to do so out of sheer rage or sorrow. (The Duke of York is the speaker of those lines.)
On the other hand, in a letter to Robert Boyle dated 1663, in The Works of the Honourable Robert Boyle (1772), Robert Hooke offers the following description of an aged “Mr. Hobbes”:
I found him to lard and seal every asseveration with a round oath, to undervalue all other men’s opinions and judgments, to defend to the utmost what he asserted though never so absurd, to have a high conceit of his own abilities and performances, though never so absurd and pitiful, &c. He would not be persuaded, but that a common spectacle-glass was as good an eye-glass for a thirty-six feet glass as the best in the world, and pretended to see better than all the rest, by holding his spectacle in his hand, which shook as fast one way as his head did the other, which I confess made me bite my tongue.
A similar instances of this usage is from an anecdote told by the Duchess of Marlborough, in William Cooke, Memoirs of Samuel Foote, Esq. (1805):
“He [the Duke of Marlborough] He came back one day from my poor misled mistress Queen Anne (I believe when he resigned his commission), and said he had told her, that he thanked God, with all his faults, neither avarice nor ambition could be laid to his charge.
“When he told me this,” continued she, “though not in a laughing humor, I bit my tongue almost through to prevent my smiling in his face.”
A similar instance of this meaning appears in Mrs. Burbury, Florence Sackville; or, Self-Dependence (1852):
Telling grievances never does any good, but almost always a great deal of harm. If you know anything ill of a man or his wife, keep it to yourself Or if you think you must tell it, bite your tongue hard for half an hour before you indulge your talking propensities.
The two instances from Cervantes involve (in Don Quixote) preferring to have inflicted the pain of biting one’s tongue to have spoken unsuitably, and (in A Dialogue Between Scipio and Bergansa) to volunteering, as an appropriate punishment, to bite one’s tongue if one commits the fault of censuring something.
A purely English occurrence of the idea of biting one’s tongue for speaking wrongly occurs even earlier than the 1719 translation of Don Quixote that Google Books finds, though the first edition of that translation may be considerably older. From William Congreve, Love for Love (1695):
Angelica. O barbarous! I never heard so insolent a Piece of Vanity—Fie, Mr. Tattle—I’ll swear I could not have believed it —Is this your Secresie?
Tattle. Gad so, the Heat of my Story carry’d me beyond my Discretion, as the Heat of the Lady’s Passion hurry’d her beyond her Reputation—But I hope you don’t know whom I mean ; for there was a great many Ladies raffled—Pox on’t, now could I bite off my Tongue.
And from the final paragraph of Book 1 of Samuel Richardson, Pamela, or Virtue Rewarded (1740):
What shall I do, what Steps take, if all this be designing!—O the Perplexities of these cruel Doubtings!—To be sure, if he is false, as I may call it, I have gone too far, much too far!— I am ready, on the Apprehension of this, to bite my forward Tongue, (or rather to beat my more forward Heart, that dictated to that poor Machine) for what I have said.