What are the relevant properties of cardinal utility functions for defining a notion of expected utility for mixed strategies in games?
Solution 1:
You are basically asking for the axioms of expected utility (note that even though it seems that way, game theory models do not require agents to be risk neutral; you can interpret the payoffs as utilities). These are commonly known as the von Neumann - Morgenstern axioms.
In the space of lotteries, the axioms on preference are:
(1) Completeness (any pair of lotteries can be compared);
(2) Transitivity (If lottery A beats B and B beats C, then A beats C);
(3) Continuity (preferences don't change "suddenly") and
(4) Independence (google it for a good definition; it basically means that common components of lotteries can be ignored while comparing them)
(5) Archimedean-ness (this is a technical property, it's related to continuity)
(3) and (5) are technical axioms, (1) and (2) are pretty standard, though often violated.
The real constraint on behaviour is (4) - Independence. Independence is violated is systematic ways (see the Allais' paradox and the Ellsberg paradox) and as such expected utility has several alternatives: non-expected utlity theory, rank-dependent utility, ambiguity averse (Gilboa-Schmeidler) utility etc.
The trouble is: none of them are as easy to use as expected utility theory. In complicated applications, simplicity is vital if you have to get anywhere. It's amazing how complicated simple games get when you move away from expected utility.
EDIT: There is a hidden sixth axiom - the space itself. Defining preference on lotteries means we're committing to representing all uncertainty with a single probability. That is to say, we're assuming that we're never unsure of probability of an uncertain event. This is taken up by the Ellsberg paradox and the ambiguity literature.