Why do we say that an obscene joke is "off-color"?

Why do we say that an obscene joke is "off-color"? Is a G-rated joke "on-color"? What color? When and how did this idiomatic expression come from?

The first definition in the OED is specific to diamond mining, where an off-colour diamond is neither pure white or another colour, which makes it of inferior value (there are quotations using the phrase from 1860 - 1968).

The next definition is more general and has two sub-definitions. The first means of a colour that's either darker, lighter, not natural, proper or acceptable (quoted 1873 - 2000). The second is to be slightly unwell, or not up to the mark, or out of order (1876 - 1997).

Finally, the third is what we're after:

Of questionable taste, disreputable; improper, vulgar; spec. (of language, jokes, etc.) slightly indecent or obscene. Cf. dirty adj. 2.

The first quotation:

1875 J. G. Holland Sevenoaks in Scribner's Monthly Mar. 582/1 Everybody invited her, and yet every body, without any definite reason, considered her a little ‘off color’.

Searching Google Books, I found some antedatings. First for diamond mining:

The Mirror of Literature, Amusement, and Instruction - Volume 5 - Page 15 - John Timbs - 1825:

The smallest flaw, or foul (as it is called) greatly diminishes the price of the diamond; and if it be tinged with yellow, brown, &c., a fault characterised by the technical term off colour, its value falls very considerably, and is frequently reduced from a third to one half.

Next, for a general use (though I'm not sure if this means "unwell" or "of questionable taste"):

Paris in '67: Or, The Great Exposition, Its Side-shows and Excursions - Page 87 - Henry Morford - 1867:

... and yet, though I have no doubt that the lady has been a little 'off color,' and though Mazeppa and the French Spy may not be exactly the thing in which we should like our sisters to 'show themselves' — yet, do you know, I am not only in love, ...

Here is the definition of off color in Chapman & Kipfer, Dictionary of American Slang, Third Edition (1995):

off color adj phr by 1875 Somewhat salacious; risqué; =BLUE: a couple of off-color jokes/ Some of his observations were a bit off color.

That same reference reports that blue in the sense of "lewd, rude, suggestive" appeared in American English by 1840.

The same sense of blue appears in England, too. Thus, Farmer & Henley, Slang and Its Analogues (1890) has this entry:

blueness subs. (common) —Indecency. Smutty talk is described as BLUE, sense 2.

Carlyle, Diderot [1840]. "The occasional blueness of both [writings] shall not altogether affright us."

And for blue sense 2, Farmer & Henley says this:

Indecent; smutty; obscene. This may be derived from the blue dress of harlots—although Hotten suggests it as coming from the French Bibliotheque Bleu, a series of books of questionable character. Books of an entirely opposite nature are said to be brown or Quakerish.

From About.com's definition of blue humor (http://comedians.about.com/od/glossary/g/bluehumor.htm):

Definition: "Blue" humor involves material that's typically considered more "adult"; it can include swearing or foul language, sexual or scatological (bathroom) humor. Most blue humor can only be heard on cable TV or satellite radio; comics rarely "work blue" on network talk shows (like The Tonight Show), mostly because of network standards. Many comics choose never to work blue, keeping their acts clean and more appropriate for all ages.

The alternative term for clean humor seems to have been white at one time. Wentworth & Flexner, Dictionary of American Slang (1960) offers this item under its entry for blue:

3 Risque; vulgar; suggesting the obscene. 1930: "...Blue gags = jokes in questionable taste." Variety. 1953: "In burlesque [comedian Red] Buttons has a reputation for being, if not lily white, at least no bluer than..." G. Millstein in NY Times Mag., Feb 22.

So putting all of this together, we have the idea that off color refers to blue, which means crude, lewd, or indecent; but white or brown indicate clean (or boring). I especially like the idea from Farmer & Henley that blue may have gotten its unsavory connotation from a particular edition of smutty French literature.