Trying to remember a synonym for a "fixed smile"

The word is rictus.

ODO has:

rictus: A fixed grimace or grin: their faces were each frozen in a terrified rictus

Vocabulary.com adds:

A rictus is a frozen, fake smile. If the star of a play finds herself overcome by stage fright, she might forget her lines and stand, trembling, her mouth twisted into a rictus.

And Wikipedia gives the medical usage:

Risus sardonicus or rictus grin is a highly characteristic, abnormal, sustained spasm of the facial muscles that appears to produce grinning.

The name of the condition, which has its roots in the Mediterranean island of Sardinia, derives from the appearance of raised eyebrows and an open "grin" – which can appear sardonic or malevolent to the lay observer – displayed by those suffering from these muscle spasms.