Origin of “on tilt”
The OED's first citation for tilt with reference to pinball machines is:
[1934 Billboard (Cincinnati) 17 Nov. 67 (advt.) Anti-tilt.]
The view among poker players is that being/going/putting "on tilt" was adopted/adapted from pinball machines. Unless use of the expression for emotion-based play in poker can be document earlier than the arrival of pinball, I would accept this explanation. The earliest example of "on tilt" used in poker that I've found is 1978 (see below).
(1) Examples of like pinball machine on tilt / titled like a pinball machine meaning uncontrolled motion /out of control / drunk:
Not Crazy—Tilted
"Tilted like a pinball machine" may be on the way to becoming a handy part of American slang. The syndicated cartoon feature, "Out Boarding House—with Major Hoople," showed the boarding house bunch ganging up on the old windbag. One of the boys ask him, "Are you tilted again like a pinball machine?" (From The Detroit News, October 28.) The Billboard, November 20, 1943
The liquor gurgled down George's gullet—and oh, boy! He lighted up like a pinball machine on tilt! A. A. Grapevine, vol. 23, p.17 and partly in the search page (1966) (Snippet Views) [1944 was the year of the first issue]
John was like a pinball machine on tilt, out of control, and he raced into the soundproof recording room. Bob Woodward; Wired: The Short Life & Fast Times of John Belushi (1984)
Then again, maybe the real issue is that I'm below in the stuffy cabin of our sailboat right now, which is acting like a pinball machine on tilt (I'm the pinball). Janna Cawrse Esarey; The Motion of the Ocean (2009)
She was sitting in full lotus pose, and her mind seemed so focused in comparison, it made me feel like a pinball machine on tilt. Christoffel Hendrickx; On My Way Home, vol. 1 (2019)
(2) Attribution/likening of poker players' on tilt state to tilt on a pinball machine:
Poker playing involves a lot of aggression, "psyching" others out and often the use of personal insults. Snippet View
The goals of these insults are to put players on tilt. Snippet View
Mike's crazy antics couldn't put him on tilt, since he was too occupied with thinking up a play to bust john... Snippet View
David Hyano in Human Behavior, vol. 8 (1978) Snippet with copyright date
The concept of tilt comes from traditional pinball machines. To keep players from damaging the machines by lifting them to alter the course of the ball, the manufacturers placed sensors inside that disabled the machine if it was violently jostled. The flippers stopped working, the lights went off, and the word "tilt" flashed at numerous places on the layout. The origin of tilt in pinball is apt because what's going on in our brain in moments of tilt is like a shaken pinball machines. When the emotional center of the brain starts pinging, the limbic system (specifically the amygdala) shuts down the prefrontal cortext. We light up...then we shut down our cognitive control center. Annie Duke; Thinking in Bets: Making Smarter Decisions When You Don't Have All the Facts (2018)
When a player goes on "tilt,” like a pinball machine, he or she stops operating and makes rash moves. (p.19)
Playing on Tilt
When you abuse a pinball machine, it will go on tilt and stop playing . The same thing will happen with many poker players. If they feel abused, they will go on tilt and stop playing their best game. James McKenna; Beyond Bluffs: Master the Mysteries of Poker (2006)
Poker players have a different vocabulary and views when it comes to the subject of problem gambling. Poker players tend not to speak of think of themselves in terms of addition. Rather poker players use the phrase "tilt." ... The term originated from the game of pinball, when frustrated players would attempt to cheat the game by tilting the pinball machine Aaron Duncan; Gambling with the Myth of the American Dream (2015)
Broadly speaking, letting emotion influence you is called being on tilt, or tilting. The term presumably comes from pinball: ram a pinball machine too hard with your hip, and the machine shuts down and you lose. C. Baldassarre and R. Burgess; The Pocket Idiot's Guide to Texas Hold'em (2006)
As the poker players would say if asked to make pinball puns, I'm on tilt. Adam Ruben; Pinball Wizards (2017)
The term tilt comes from a pinball tilt. A player on "tilt" is one who's playing very badly as a result of some emotional trauma. Gary Carson; The Complete Book of Hold 'Em Poker (2001)
They go "on tilt.” Sort of like a pinball machine, except with pinball it only costs you a quarter. Daniel Kimberg; Serious Poker (2002)
It is enough to make good players go on tilt. What is tilt? Did you ever play pinball? Remember what happens if you jostle the pinball machine too much? You cause it to shut down. Ashley Adams; Winning 7-Card Stud (2003)
From Etymonline:
tilt (n.1)
Pinball machine sense is from 1934
The 'tilt' mechanism was invented in 1934 as a direct answer to the problem of players physically lifting and shaking the games.
So I believe the psychological meaning of on tilt in poker (also chess and esports) originates from the physical meaning of (on) tilt in pinball. There aren't any instances of the psychological sense of tilt being used in poker before 1934, so this seems to be the only logical explanation.
"Tilt" originates from pinball tables, where altering the slope of the table would affect the path of the ball. As this is essentially cheating, pinball tables were designed with a tilt sensor, which when triggered would disable all scoring mechanisms (and usually the flippers as well), forfeiting any further action with that ball.
A player that's frustrated may shove the table in anger, triggering the tilt sensor. (Gentle nudging was a common tactic, if one was familiar enough with a particular pinball table to know the sensitivity of the sensor.) So the expression "on tilt" started to be used to mean a player who's lost his cool.
Greg B from Gaming Stack Exchange
I didn't know the idiom myself, but a bit of googling soon found...
Tilt = Anger + Bad Play (The Mental Game of Poker, 2011)
The definition of tilt as defined by conventional poker wisdom is too broad. Sometimes tilt just means playing badly; but it also can mean playing badly because of winning, or playing too loose, conservatively, ...
...followed by several pages going into detail about different types of "tilt", within which the specific usage [to be] on tilt occurs repeatedly.
I looked at several other instances of the sequence were on tilt in Google Books, and most of them were obviously related to either poker playing specifically, or gambling / risk-taking sport in general, so an origin in gambling looks likely to me. But I'd say it's essentially a fairly transparent metaphoric usage implying that a player has been knocked off balance, destabilised ("lost his cool"), and is thus not playing well / sensibly.
EDIT Seeing various answers and comments relating to tilting at windmills, rushing full-tilt, tilting pinball tables, I think it's worth reproducing an actual dictionary definition of this specific sense...
on tilt
In a reckless or rash state; acting without proper care, attention, or consideration.
Originally used in reference to poker players who begin making reckless decisions after experiencing either good or bad luck. Primarily heard in US.