How do I swap Ctrl and Alt in just the Terminal?

iTerm2 will let you remap your modifier keys. This has the effect of just remapping them for terminals. Bonus: it's also a better terminal program for OS X than the built-in Terminal.app that ships with OS X.

You can alter the modifier keys on a per-profile basis as well. That lets you switch profiles for terminals depending on what you're doing with the terminal so you get the correct behaviour for local and remote sessions.

You may find "Karabiner" app to fit your needs. I have used this app for similar reasons as to swap keys with modifiers.

You won't find a per app setting in the preferences. You will, however, find a setting where you can switch the left 'command' key with the left 'control' key ONLY in certain apps (ie. Terminal, Chrome, Emacs, etc.)

You can do a filtered search for "terminal" or "Command_L" to specify the effect you are seeking.