Etymology of the phrase "chicken out"
The OED says this verb chicken is slang of US origin with a first quotation from 1943 (I. Wolfert, Torpedo 8):
I just wanted to..make sure you weren't chickening out on me.
They say this is a revived form coming from a noun chicken for one who is as timorous or defenceless as a chicken, used at least as early as 1616, and cite Shakespeare (Cymbeline 1623):
Forthwith they flye Chickens, the way which they stopt Eagles.
Chickening Out may come from 1864 Union Army enlistment in which a chicken was provided to each person who enlisted. He would take the chicken home, clean, dress, and cook it for dinner- no refrigeration in those days. The next day he came back to ship off for the Union Army.
Should he not come back his name was printed in the local paper-very shameful to the family name - not like today. A relative with the same family name could fulfill the Army contract by enlisting instead of the original person. The Union Army didn't care as long as they got someone of worth.
This occurred to my Great Great Grandfather in August 30th of 1864. He was a farmer in the southern Adirondacks of NY, 39 years old. He left his wife and 4 children to cover a contract to spare the family name the shame and disgrace from a nephew who "chickened out".
At 39, he was not exactly front line material. The Union Army made him a teamster driving supply wagons. Company E, 115th regiment of New York Volunteers. He died of wounds he received at the front in Richmond VA from a sharp shooter on Oct 27 1864. His place of burial has never been identified.
If you see a fight between two male chickens the loser will runaway with a low neck position is a sign of submission. Hence chickened out?
J.E. Lighter, Random House Historical Dictionary of American Slang (1994) has a couple of citations for "chicken out" that predate the OED's first instance from 1943 (cited in Hugo's answer):
chicken out v. to back out, as from fear; renege. Also (in more recent use) chicken. [First two cited instances:] 1934 Weseen Dict. Slang 177: College {Slang}...chicken out—To fail {sic}. 1941–42 Gach In Army Now 142: Old Uncle Sam promised to let me go back ...in a year. He can't chicken out on me now.
Throughout its period of use in U.S. slang, "chicken out" has coexisted with the adjective chicken in the sense of "pusillanimous." Here is the start of Lighter's entry for chicken in that sense:
chicken adj. 1. cowardly; afraid. [First three cited instances:] 1933 D. Boehm & E. Gelsey Jimmy Dolan (film): Ain't turnin' chicken, are ya? 1929–34 Farrell Judgment Day 532: He was the skinny, dark-haired punk around the corner who was so chicken, wasn't he? 1939 They Made Me a Criminal (film) Boy, is that guy chicken.
A mere 125 years ago, chicken seems not to have carried any sense of cowardice or of failure to follow through. J.S. Farmer & W.E. Henley, Slang & Its Analogues (1891) lists meanings for chicken that range from "a pint pot" (in thieves' cant) to "a child or young person" to an anticipated benefit or success (in the phrase "don't count your chickens before their hatched") to "a poulterer; also a sportsman's term for anyone shooting immature game"(in the term "chicken-butcher") to "any fare out the common, and also to show of any kind" (in the term "chicken-fixings"). And that's it.
The common (until it was banned, in 1835) spectacle of cockfighting in England may help explain why, in Dombey and Son (1846–1848), Dickens identifies a renowned pugilist and athletic adviser to the naive but good-hearted Mr. Toots as "the Game Chicken":
In this delicious abode [his "choice set of apartments"], Mr. Toots devoted himself to the cultivation of those gentle [sporting] arts which refine and humanise existence, his chief instructor in which was an interesting character called the Game Chicken, who was always to be heard of at the bar of the Black Badger, wore a shaggy white great-coat in the warmest weather, and knocked Mr. Toots about the head three times a week, for the small consideration of ten and six per visit.
It turns out that "the Game Chicken" was the nom de guerre of the British boxing champion Henry Pearce in the early 1800s, as described in History of British Boxing, From Fig and Broughton to the Present Time, chapter 2, in The Sportsman's Magazine (July 19, 1845). It should be clear from this example that chicken bore no hint of cowardice or reneging in the 1800s; those associations seem to have developed in the first half of the twentieth century.