When and where did "clam" come to mean a missed note in a musical performance?
Solution 1:
The earliest coverage of clam in the sense of "mistake or misplayed musical note" that I've been able to find is in Harold Wentwoth & Stuart Flexner, Dictionary of American Slang, first edition (1960):
clam n. ... 5. a mistake ; a boner 1952: "Radio men speak no worse English than the general average of American professional men, and ... on the average there are not really so horribly many out-and-out clams coming over the air." Letter in N.Y. Her[ald] Trib[une], Apr. 3, 24/5.
J.E. Lighter, Random House Historical Dictionary of American Slang (1994) cites the 1952 instance reported in Wentworth & Flexner as its first recorded occurrence of the usage, but then adds a slightly later occurrence, from 1955, and still later ones from 1964 and 1991:
clam n. ... 5. Entertainment Industry. a mistake, BONER; (Jazz) a misplayed note. 1952 in DAS. 1955 Down Beat (Nov. 30) 47: I'd say that was a band that doesn't work together regularly ... because there were a few clams in the ensemble. 1964 New Yorker (Apr. 25) 195: Clams proliferate. 1991 N.Y. Times (Dec. 1) H 33: Bing Crosby ... always said, 'Leave the clams in, let 'em know I'm human.'"
Robert Chapman & Barbara Kipfer, Dictionary of American Slang, third edition (1995) reports that the expression goes back to the 1940s but does not cite any historical examples:
clam ... 4 n 1940s jazz musicians A wrong or sour note; =CLINKER
Coincidentally (or not), Chapman & Kipfer also notes that clambake had a special meaning in jazz circles, starting in the 1930s:
clambake ... 2 n 1930s jazz musicians =JAM SESSION
So if clam originated as a jazz term in the 1940s, rather than more broadly as an U.S. entertainment industry term by the early 1950s, a clam (wrong note) may have referred to the sort of mistake that a jazz musician would expect to encounter more frequently at a clambake (jam session) than at a formal, paying performance.
Clarence Major, Juba to Jive: A Dictionary of African-American Slang (1994) lends some support to this theory in its entries for clam and clambake, both of which cite entries that originated in Robert Gold, A Jazz Lexicon (1964):
Clam n. (1940s–1950s) in jazz, a misplaced note. ([Jazz Lexicon], p. 59) Example: "The band isn't in good shape tonight. Too many clams." ...
Clambake n. (1930s–1950s) an "ad lib {jam} session ... not in the groove"; a "swing" or jam session; (1950s) a jazz or rhythm-and-blues musical affair that doesn't come off well. ([Jazz Lexicon], p. 58) ...
And H. Brook Webb, "The Slang of Jazz," in American Speech (1937) confirms that clambake was used to refer to a jam session in the 1930s [combined snippets]:
CLAMBAKE. Same as jam session.
...
JAM SESSION. An after hour's gathering of musicians of the hot persuasion from different bands, for relaxation, refreshment, and to jam a few numbers with everyone cutting a lick or two when the spirit moves.
I'm not entirely sold on the clam-by-way-of-clambake explanation, in part because it doesn't persuasively indicate why a word chosen presumably for its pleasant associations with relaxed good times (clambake) should have yielded a derogatory term for faulty musicianship (clam), and in part because it isn't clear to me that clam originated as jazz talk rather than in the wider world of U.S. entertainment.
Nevertheless, I think that 1930–1952 is very likely the approximate origin period for both clambake and clam in the relevant senses. And any better explanation of the origin of clam as "mistake" should devote at least a little time to explaining why the jazz terms clam and clambake are not related.
Update (May 21, 2021):
After posting this answer, I bought a copy of Robert Gold, A Jazz Lexicon (1964), which (as noted in my original answer) Clarence Major cites in his entries for clam and clambake. Gold's entries for the two terms are quite a bit more detailed than Majors's and are worthy of reproduction at full length:
clam n. {poss. partly from being alliterative with its older synonym clinker [which Gold views as "onomatopoeic"]; more prob. shortened form of the derogatory sense of clambake; current since c. 1950; see also GOOF} A misplayed note; also, for a rare verb use, see 1961 quot. — 1955 Down Beat, 30 Nov., p. 47. I'd say that was a band that doesn't work together regularly ... because there were a few clams in the ensemble. — 1961 Down Beat, 2 Feb., p. 30. Hubbard sounds positively uncomfortable and clams in royal style at the beginning.
clambake, n. {by analogy with the standard sense; current c. 1930–c. 1938 in an approbative or a neutral sense, but increasingly since c. 1938 in a pejorative sense} See 1952 and first 1955 quots. — 1937 American Speech, Feb., p. 46. clambake: same as jam session. — 1938 Cab Callaway: Hi De Ho, p. 16. clambake: ad lib session, every man for himself, a jam session not in the groove. — 1949 Music Library Association Notes, Dec., p. 41. clambake: gathering of hot musicians. Also used in a derogatory sense to mean an affair that does not come off well. — 1952 A History of Jazz in America, p. 350. clambake: earlier used synonymously (and honorifically) with "jam session," later descriptive of an improvised or arranged session which doesn't come off. — 1955 The Encyclopedia of Jazz, p. 346, clambake: Originally, a jam session; currently, an unsuccessful, disorganized session. — 1955 Hear Me Talkin to Ya, p. 265. Everybody got kind of half-high and it ended up in a clambake.
Gold's discussion and cited quotations make a strong circumstantial case for clam to have emerged from the pejorative sense of clambake—a sense that I was not aware of when I wrote my original answer. The timing is certainly right for such an origin, since clambake had evidently acquired negative connotations at least as early as Cab Calloway's Hi De Ho glossary, which was published in 1938, and the paper trail for clam in the relevant sense begins about a dozen years later.