Literal meaning and etymology of “off” in “bake off”, “dance-off” etc

For its part, the OED attributes all these compounds to variations of play-off, which in sports held this early meaning cited in 1895:

A match or rematch played to decide a draw or tie.

However, in attributing these variations like "bake-off" and "cook-off" to "play-off," the OED is opting not to attribute the phrases to the earlier terms cited in Edwin Ashworth's answer, face-off which held an earlier meaning in hockey, and stand-off, which first meant a tie in a game of cards, which might not fall into the same realm of meaning as "bake-off" or "cook-off" or "play-off," but certainly seems related.

Thus, if a man bets on the ace and deuce, and the ace comes to his side, and the deuce to the dealer's side, it is a stand-off, and neither wins.

• 1843 J. H. Green Exposure Arts & Miseries Gambling 187

Ultimately, I find the OED's conclusion most satisfactory, as a "bake-off" or "dance-off" are, in essence, tournament-style competitions similar to a "play-off."

As a transitive verb, "play-off" is cited earlier than as a noun, in 1870:

On the tie being played off, Sir Robert and Mr. Anderson again tied.

• 1870 C. MacArthur Golfer's Ann. 118

Only later, first cited in 1932, did "play-off" come to mean:

(Each of) a series of games, matches, or contests played to decide a championship, competition, etc.; (also) a preliminary series of matches to decide which individuals or teams qualify for a championship or competition.

As in this citation:

The play-off for the Middle Atlantic League baseball title will start Wednesday.

• 1932 Sun (Baltimore) 6 Sept. 14/2

And it wasn't long until people were doing more than just "playing" off.

There will be a grand cook-off with five sectional winners competing.

• 1936 Port Arthur (Texas) News 3 May d4/3

Although this hardly impacts the conclusion of this answer, "cook-off" can be antedated beyond the OED's citations. This 1931 newspaper clipping describes a "Cooking Contest" as a "cook off."

• 1931 - The Brooklyn Daily Eagle (Brooklyn, NY) 01 Oct.

The following extract from World Detective appears to shed more light on the usage of off in play off:

Meanwhile, a slightly different sense of “off” had been dumped into the mix. In 1870 the phrasal noun “play-off” first appeared in print meaning a game played to decide a tie at the end of a previous game (“The tie game of yesterday was played off to-day,” 1880). This “play-off” invoked a very old sense of “off” meaning “exhaust or finish completely” (as in our modern “finish off”).

It wasn’t until 1932 that “play-off” came to mean (first in the US, of course) “a series of games, matches, or contests played to decide a championship, competition, etc.” (Oxford English Dictionary).

The precedents of “face-off” and “play-off” subsequently served as models for all the little “X-offs” you’ve noticed, from “cook-off” (1936) to “bake-off” (1949) to the more recent “dance-off,” “sing-off,” etc. These terms all employ both the “confrontation or contest” sense and the “finish” sense (in that there is only one winner) of “off.”

I found an early instance of "play-off" as a noun in "That Three-Mile Run on the Fashion Course: Tar River Showing His Heels Again!" in the New York Clipper (June 25, 1859), about a horse race:

JUNE 17, 1859 The intended play-off, or rather return match, between the horses Tar River and Bill Cheatham, took place as announced, on the 17th, notwithstanding a muddy track and torrents of rain. The Fashion Course kept "open house" on this occasion, the rain making the company "quite select." Bill having been victorious in the two-mile run on the 14th, so far met the general expectation of public judgment that Tar River was considered equally certain in the result of the three mile test. Bill's Albion blood could hardly be counted upon for endurance, its general reputation being rather more on the dashing order.

The reporter chooses to call the follow-up race between Tar River and Bill Cheatham over a three-mile course a "play-off" for reasons he doesn't explain. The previous race (three days earlier) was over a two-mile course, and the reporter seems to say that Bill Cheatham won it. In any event, Tar River won the return match over the longer course, leaving the "play-off" picture without an overall winner. My impression is that "play-off" here means nothing more precise than "contest."

Something more seems to be at stake in this report on a billiards tournament, also from the New York Clipper (November 29, 1873):

The telegraph recounts that in the play-off game between Garnier and Ubassy the latter felt aggrieved by two decisions given by the referee and was with difficulty prevailed upon to continue the game. The decisions were purely as to questions of fact, it is proper to state, and there could be no appeal from them, unless it could be shown that the referee had good reason to entertain a bias; and as to this it should be explained that Ubassy was claimed as Chicago's representative in that tourney, and that the referee was himself a Chicagoan—Al. Smith, well known in sporting circles. The presumption is, therefore, that his rulings were correct.

In this case, the "play-off game" may be a single contest, but it occurs in the context of a tournament, a frequent setting for playoffs in the modern sense. In fact, billiards seems to have used the term play-off commonly in the middle 1870s. Other New York Clipper articles from February 28, 1874, May 2, 1874, and January 23, 1875 all refer to play-offs for finishing positions in a multiple-player game of pool or in a billiards tournament. Additional matches from 1875 in the same newspaper discuss play-off contests in eucher and other card games involving wagers.