How do you make it obvious you are on a production system?

The red prompt is a good idea, which I also use.

Another trick is to put a large ASCII-art warning in the /etc/motd file.
Having something like this greet you when you log in should get your attention:

 _______ _    _ _____  _____   _____  _____            
|__   __| |  | |_   _|/ ____| |_   _|/ ____|     /\    
   | |  | |__| | | | | (___     | | | (___      /  \   
   | |  |  __  | | |  \___ \    | |  \___ \    / /\ \  
   | |  | |  | |_| |_ ____) |  _| |_ ____) |  / ____ \ 
   |_|  |_|  |_|_____|_____/  |_____|_____/  /_/    \_\


 _____  _____   ____  _____  _    _  _____ _______ _____ ____  _   _ 
|  __ \|  __ \ / __ \|  __ \| |  | |/ ____|__   __|_   _/ __ \| \ | |
| |__) | |__) | |  | | |  | | |  | | |       | |    | || |  | |  \| |
|  ___/|  _  /| |  | | |  | | |  | | |       | |    | || |  | | . ` |
| |    | | \ \| |__| | |__| | |__| | |____   | |   _| || |__| | |\  |
|_|    |_|  \_\\____/|_____/ \____/ \_____|  |_|  |_____\____/|_| \_|


 __  __          _____ _    _ _____ _   _ ______ 
|  \/  |   /\   / ____| |  | |_   _| \ | |  ____|
| \  / |  /  \ | |    | |__| | | | |  \| | |__   
| |\/| | / /\ \| |    |  __  | | | | . ` |  __|  
| |  | |/ ____ \ |____| |  | |_| |_| |\  | |____ 
|_|  |_/_/    \_\_____|_|  |_|_____|_| \_|______|

You could generate such a warning on this website or you could use the figlet command.

Like Nicholas Smith suggested in the comments, you could spice things up with some dragons or other animals using the cowsay command.

Instead of using the /etc/motd file, you could also call cowsay or figlet in the .profile file.

Not quite the same thing, but this web site recommends having your developers wear a pink sombrero when making changes to production systems. You could probably have a similar rule for sshing into them.

The biggest I've used is a discrete naming-scheme where prod-systems are named obviously different than test/dev instances. This makes the "Username@Hostname: " style prompt visibly different. And by obvious I mean more than just different words, different formats too:

example: PRD-WEB001 vs DEVEL-BOB-WEB001

This has several things going for it:

• The extra hypenated block makes it a set-of-three instead of a set-of-two.
• The first of the set is a different length.
• The overall length of names is markedly different, which makes the command-line spacing different relative to each other and other text in the window.

And best of all, it doesn't require special terminal-configs for production just to avoid Oops errors.

In my experience, you want something that is a constant reminder of where you are. Login-methods like riddles are good for about 10 seconds, until you forget which window is which. All it takes is to do an ls in the wrong directory to scroll the ominous login-banner out of view, bury the terminal window under a browser window while googling something, alt-tab back to the wrong window and mayhem ensues. Best to have some constant visual cue like a significantly different command-prompt.

One thing you need to keep in mind is that this needs to be a persistent reminder, not just an indicator at login time. Very often, someone will have several shells running at the same time in different tabs and move between them. Some will be dev, some production. So when you are running a command, you need to have an indicator at that point. So having a special prompt is the best method, in my experience, with a modified title/tab bar being a nice complement to this for finding the right window/tab easily.

So I'd recommend having a colored prompt (red being the obvious choice) and all caps for the hostname, with similar behavior for the user (privileged vs. non-privileged) as your prompt. Some examples:

Usually something like

set prompt =  "%{\033[1;44m%}`whoami`@`hostname -s`#%{\033[0m%} "` 

in your shell startup file. This one is for the blue. Replace the 44 with 41 fir red, and 42 for green. Other colors and wild patterns available too.