What is the origin of the phrase "when push comes to shove"?

"When push comes to shove" means "as a last resort" or "if absolutely necessary". Does anyone know why the phrase came to be used in this way?

According to this site, this term comes from rugby, where, after an infraction of rules, forwards from each team face off and push against one another.

I'm not sure about this rugby scrum origin theory. I found a snippet of a 1981 William Safire column in New York Times Magazine article that indicates this guess was put forth by an AJ Gracia of Southbury, Connecticut. It goes on to call it an "offbeat etymology."

Etymonline has the phrase dated from 1958, but with no mention of rugby. I found the phrase used over a decade earlier (1947) in the English translation of Haitian Jacques Romain's Masters of the Dew done by Langston Hughes and Mercer Cook. Again, no rugby context:

It would be interesting to know what original French phrase was translated as such. Hughes went on to use the phrase again in Simple Takes a Wife, 1953—the next reference I can find in print.

Edit:

Safire wrote another column on the phrase in 1997 which drops the mention of rugby altogether. He concludes "a black-English origin for the phrase is pretty likely" and cites a 1954 example, four years earlier than OED's earliest reference (1958). He also dug up this more plausible origin of the phrase:

Other evidence there of the phrase's black origin is a recollection from Norman Pierce of Jack's Record Cellar in San Francisco of Shove Day, or Bump Day, the traditional Thursday off for domestic servants in the 1920's, ''on which blacks 'accidentally' jostled whites in public places, railways, streetcars, etc.''

Edit #2:

I just found another antedating of the phrase. These are from Black Thunder by Arna Wendell Bontemps (close friend of Langston Hughes), 1936: