What line do they refer to in the idiomatic expression "on the line"?

Dictionary coverage of 'on the line'

Christine Ammer, The American Heritage Dictionary of Idioms (1995), connects "on the line" with "lay on the line," which has yet another variant form, "lay it on the line." Here is Ammer's discussion:

lay on the line 1. Make ready for payment, as in They laid hundreds of thousands of dollars on the line to develop new the new software. [c. 1900] 2. lay it on the line. Speak frankly and firmly , making something clear. The professor laid it on the line: either hand in the term paper or fail the course. [c. 1920]

But Christine Ammer, The Facts on File Dictionary of Clichés, second edition (2006), has a slightly different take on the phrase "lay it on the line":

lay it on the line, to To speak frankly. This Americanism of the early twentieth century originally meant to hand over money (from about the 1920s). However, by mid-century it meant to speak plainly or categorically, and in the 1960s acquired still another sense, to lay something on the line, meaning to put that thing at risk (as in, "The Marines laid their lives on the line").

J.E. Lighter, Random House Historical Dictionary of American Slang (1997) has three definitions for phrases that include the phrase "on the line":

lay {or put} it on the line {fr the gambling sense 'to put money on the line {space on a craps table for placing chips}'} 1. to put down or hand over money. Now colloq. or S[tandard] E[nglish]. [First cited occurrence:] 1929 D[amon] Runyan, in OED2: My rent is way overdue...and I have a hard-hearted landlady....She says she will give me the wind if I do not lay something on the line at once. ...

2. to explain; speak one's mind firmly and directly. Now colloq. [First cited occurrence:] 1944 Butler & Cavett Going My Way (film): Like Tony says, I'm gonna lay it on the line.

on the line engaged in prostitution, esp. in a red light district. [First cited occurrence:] 1910 in Roe Prodigal Daughter 83: Oh if everybody did know the awful shame and degradation of such a life there would surely be very few girls "on the line."

'Lay it on the line' as 'bet or spend money on it'

The earliest Google Books match for "lay/put it on the line" in the sense of "bet on it" is from "This 'Mere Mechanical Toy' Was Fifty Feet Long!" in Popular Science (December 1931):

...how in the name of common sense could the monster be running round with a clothed young lady in its mouth? It just doesn't stand to reason unless I'm getting cock-eyed in my old age. And anyway, if by some freak of nature a girl had strayed into the dinosaur age, it's a perfectly safe bet of a million to one that she'd have been clothed in just exactly nothing or quaintly wrapped in a rapidly decomposing piece of hide. Am I right? You can put it on the line that I am.

And from E.A. Batchelor, "Gasoline-Buggy Beginnings," in The Rotarian (March 1934):

So that good old Anglo-Saxon custom of trying to tax anybody to death if he seems to be getting along was instituted. Most of the roads were privately controlled in those days; turnpikes upon which appeared at irritating intervals toll gates where the traveller "laid it on the line" until it hurt. The steam carriages were just made to order for the toll-gate boys. The steam carriages were just made to order for the toll-gate boys.

And from Raymond Chandler, The Big Sleep (1939):

"Does there have to be something else?"

"Yes."

She stared at me a little puzzled. "There is. The woman said there was a police jam connected with it and I'd better lay it [the blackmail money] on the line fast, or I'd be talking to my sister through a wire screen."

'Lay it on the line' as 'speak plainly'

The "spoke bluntly" sense of "lay it on the line" first appears in "Chi Feels News Segs To Hold Continuous Sock Audience Pull," in The Billboard (September 29, 1945):

Occasion was forum held by Chicago Radio Management Club. Webs don't figure on losing many sponsored programs, tho some paring of commentators was envisioned, to be replaced by on-the-spot boys—both local and national.

Here's the way the boys laid it on the line: [Quotations from four network news programmers omitted.]

From "What Businessmen Themselves Foresee," in Kiplinger's Personal Finance (July 1949):

Hundreds of letters from working businessmen have recently been received by the editor of this magazine. The men laid it on the line—freely, frankly, without any pose. They told how their own business was faring, and what they foresaw for the future. They were practical and down to earth. Here is a brief report on what they thought and said.

And from "Our Enemy Is Confident; and He Tells Us, 'This Is War'," in Life magazine (December 18, 1950):

Our Communist enemy laid it on the line with a sureness and confidence which went beyond the familiar tone and language of mere propaganda. Probably the most meaningful words of the week were spoken at Lake Success by Andrei Vishinsky, the Soviet Foreign Minister. He was speaking of a mild little resolution, put up by the U.S. and five other powers. It simply asked the Chinese Communists to withdraw their troops from Korea.

Conclusions

According to Random House Historical Dictionary of American Slang, both senses of "on the line" that the OP asks about derive from the same source: a reference to the physical line on a craps table where gamblers place their chips in making a bet. That dictionary finds examples of the two senses of the phrase from 1929 (in the sense of "to put down or hand over money") and 1944 (in the sense of "to speak one's mind one's mind firmly and directly"). Google Books searches turn up instances of the phrase in each sense starting in 1931 and 1945, respectively.

None of the other reference works I consulted offers any documentary evidence contradicting the RHHDAS's findings.