How do the Spy's masks work?

Solution 1:

If you're taking the Spy-masks (or really, anything in Team Fortress 2) at face value (bah-dum-tiss), then you really shouldn't. TF2 is a game that prides itself on over-the-top, tongue-in-cheek, absurdist humour:

Surreal humour (also known as absurdist humour) is a form of humour predicated on deliberate violations of causal reasoning, producing events and behaviours that are obviously illogical. Constructions of surreal humour tend to involve bizarre juxtapositions, non-sequiturs, irrational or absurd situations and expressions of nonsense.

Wikipedia (Emphasis mine).

The masks are just a bit of hilarious flavor for a team to know which class a friendly Spy disguised as. It's supposed to be a joke, either aimed at the Spy's 'terrible' attempts to disguise himself, or at the enemy team for falling for such a ridiculous disguise (or both).

However the Spy does in fact disguise himself well: pretty much identical to a member of the other team, (from their perspective). Therefore, the masks themselves do not portray an accurate representation of the Spy's skill at disguising.

Solution 2:

The device used by the Spy in order to disguise is known as The Disguise Kit. This kit contains a few sticks of his iconic cigarettes, along with a not often mentioned piece of technology known as the Spytron 3000:

Although not much is revealed about it, this device is a technological breakthrough in infiltration, giving the Spy this great power of tricking the enemy team into seeing whoever the Spy wants them to see.

Seeing as how the disguise is achieved technologically, and based on the visual effects of cloaking (see Meet the Spy at 2:09), the most likely cause of the disguise is in the form of a holographic projection. As for why the teammates can see the the masks and "undisguised body", we can only speculate: perhaps this is caused by an adjustment to the hologram, tuned to a frequency known only to the own team, or perhaps it's a representation of a close familiarity with the Spy by his own team members (I would recognize that walk anywhere; that's my friend, Monsieur Blu Spy, not an enemy Pyro).

But then what are the masks used for? And if the masks aren't small enough to be kept in his Disguise Kit, then where does the Spy then get them from you ask? Don't ask. Perhaps the same place weapons in the secondary slots when they aren't holding on to them. (which might explain why the running speed of the Heavy Weapons Guy is so slow even when he is holding a smaller weapon...)

Solution 3:

The nine men (or 8 men and 1 woman, depending on who you ask) in Team Fortress 2 are, in fact, dead and trapped in Limbo. They fight out the same battles over and over from different perspectives for all eternity, never guessing their real situation. They don a wide variety of hats to try to prove to themselves that they're not the same person as their counterpart on the other team. They make up holidays to try to track time, even though time stands still for them. They've long since forgotten their real names, remembering only the role they fill.

These poor souls try hard to ignore the truth. So if they see a picture of their teammate on a paper mask, they'll believe the wearer is in fact their teammate.