What's the opposite of nominal in the astronaut sense?

Solution 1:

If something is nominal, then it is acting as expected, i.e., all measurements of the system are coming back in the normal/acceptable range. It's shorthand for all measurements of system X are operating as expected.

If something is not nominal, it does no good to say "not nominal", rather, one would say what is actually wrong. For instance, if fuel tank pressure was below its safe/acceptable value, they would report that the fuel tank pressure is low. That's why there appear to be a million dials or so in cockpits in addition to a computer interface. When something is off, exactly what is off is reported, and that is what the pilot / astronaut will relay.

Solution 2:

I don’t know what they actually say (did a bit of Googling for Apollo records, but couldn't find what I wanted there). As “nominal” means “that has its nominal (aka normal) value”, the common word that comes to mind is abnormal.

In writing, however, it is perfectly fine to refer to such a situation as off-nominal. This term is found in plenty of technical reports and manuals of aircraft. See also, for example, this NASA webpage: Emergency, Abnormal, and Off-nominal Situations Study.