How do I set a default language (input source) for an application?

A workaround I'm using is to make Esc always switch back to English within MacVim. There is a very nice app called KeyRemap4MacBook, and you can use something like below as its private.xml configuration to let MacVim behave in such way. MacVim might start, or be in a different input source, but you can get it right with a single keystroke, which must be pretty familiar to your fingers already. Moreover, this is extremely helpful if you're switching back and forth from input mode to/from normal mode in MacVim, entering and editing non-English text.

<root>
<item>
    <name>In Vim, Make Escape key also Switch to English IM</name>
    <identifier>private.vim_esc_also_switches_to_english_inputmode</identifier>
    <only>VI</only>
    <autogen>__KeyToKey__ KeyCode::ESCAPE,
    <!---->               KeyCode::ESCAPE,KeyCode::VK_CHANGE_INPUTMODE_ENGLISH </autogen>
</item>
</root>

See also: my private.xml in use.

There should be an option in the settings pane for input where you can select if you want the same global or on application basis.

Edit: It only have "Use same one in all documents" and "Allow a different one for each document"

Not sure that works for applications.