Application Crashes With "Internal Error In The .NET Runtime"

We have an application written against .NET 4.0 which over the weekend crashed, putting the following message into the event log:

Application: PnrRetrieverService.exe Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 791F9AAA (79140000) with exit code 80131506.

This is on a Windows Server 2003 R2 Standard Edition box. Googling this error hasn't turned up anything pertinent. For example, this isn't occurring in VS Studio, but instead on a production box; when the service was eventually restarted, it experienced no further problems.

How does one go about diagnosing a bug in the .NET Runtime?

with exit code 80131506

That's a nasty one, ExecutionEngineException. Starting with .NET 4.0, this exception immediately terminates the program. The generic cause is corruption of the state of the garbage collected heap. Which in turn is invariably caused by unmanaged code. The exact location in code at which this exception is raised isn't helpful, the corruption usually occurred well before the damage is detected.

Finding the exact cause for this is going to be difficult. Review any unmanaged code your service might be using. Suspect environmental problems if there is no obvious candidate, misbehaving malware scanners are notorious. If it repeats very poorly then suspect hardware problems like soft RAM errors.

A bug in the concurrent implementation of the Garbage Collection on x64 .Net 4 can cause this as stated in the following microsoft KB entry:

ExecutionEngineException occurs during Garbage Collection

You should first make a deep minidump exploration to be sure that the problem occured during a Garbage collection.

The minidump location can usually be found in a Windows Error Reporting entry in the event log following the crash entry. Then, have fun with WinDbg !

The latest documentation on the use of the <gcConcurrent/> configuration element, to disable concurrent or (in .NET 4 and later) background garbage collection, can be found here.

I've experienced "internal errors" in the .NET runtime that turned out to be caused by bugs in my code; don't think that just because it was an "internal error" in the .NET runtime that there isn't a bug in your code as the root cause. Always always always blame your own code before you blame someone else's.

Hopefully you have logging and exception/stack trace information to point you where to start looking, or that you can repeat the state of the system before the crash.

For those arriving here from google, I've eventually come across this SO question, and this specific answer solved my problem. I've contacted Microsoft for the hotfix through the live chat on support.microsoft.com and they sent me a link to the hotfix by email.