Configure Visual Studio to show error messages in English

Deinstall the .NET Framework xxx Language Pack. (xxx = boring message language)

The best way would be to use this code in your application entry method

if (Debugger.IsAttached)
    CultureInfo.DefaultThreadCurrentUICulture = CultureInfo.GetCultureInfo("en-US");

It will force english messages not only in exceptions caught and displayed in the application but inside the IDE as well

Because you cannot force your users to use english language versions of Windows when performing some initial tests on premises you may have find this useful.

Under Tools/Options/International settings. I have an option to change the language from "Same as Microsoft Windows" to "English" (Visual Studio 2008 in case it makes any difference). If you don't have English in there then I'm not sure how you add more languages...

Edited to add:

Since you are talking about application exceptions you need to change the culture of the application you are dubugging, you can do that by following this or if it isn't an option to change the culture for the whole app this question has some ideas for only changing culture when exceptions are thrown.