Weird MSC 8.0 error: "The value of ESP was not properly saved across a function call..."

We recently attempted to break apart some of our Visual Studio projects into libraries, and everything seemed to compile and build fine in a test project with one of the library projects as a dependency. However, attempting to run the application gave us the following nasty run-time error message:

Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function pointer declared with a different calling convention.

We have never even specified calling conventions (__cdecl etc.) for our functions, leaving all the compiler switches on the default. I checked and the project settings are consistent for calling convention across the library and test projects.

Update: One of our devs changed the "Basic Runtime Checks" project setting from "Both (/RTC1, equiv. to /RTCsu)" to "Default" and the run-time vanished, leaving the program running apparently correctly. I do not trust this at all. Was this a proper solution, or a dangerous hack?

This debug error means that the stack pointer register is not returned to its original value after the function call, i.e. that the number of pushes before the function call were not followed by the equal number of pops after the call.

There are 2 reasons for this that I know (both with dynamically loaded libraries). #1 is what VC++ is describing in the error message, but I don't think this is the most often cause of the error (see #2).

1) Mismatched calling conventions:

The caller and the callee do not have a proper agreement on who is going to do what. For example, if you're calling a DLL function that is _stdcall, but you for some reason have it declared as a _cdecl (default in VC++) in your call. This would happen a lot if you're using different languages in different modules etc.

You would have to inspect the declaration of the offending function, and make sure it is not declared twice, and differently.

2) Mismatched types:

The caller and the callee are not compiled with the same types. For example, a common header defines the types in the API and has recently changed, and one module was recompiled, but the other was not--i.e. some types may have a different size in the caller and in the callee.

In that case, the caller pushes the arguments of one size, but the callee (if you're using _stdcall where the callee cleans the stack) pops the different size. The ESP is not, thus, returned to the correct value.

(Of course, these arguments, and others below them, would seem garbled in the called function, but sometimes you can survive that without a visible crash.)

If you have access to all the code, simply recompile it.

I read this in other forum

I was having the same problem, but I just FIXED it. I was getting the same error from the following code:

HMODULE hPowerFunctions = LoadLibrary("Powrprof.dll");
typedef bool (*tSetSuspendStateSig)(BOOL, BOOL, BOOL);

tSetSuspendState SetSuspendState = (tSuspendStateSig)GetProcAddress(hPowerfunctions, "SetSuspendState");

result = SetSuspendState(false, false, false); <---- This line was where the error popped up. 

After some investigation, I changed one of the lines to:

typedef bool (WINAPI*tSetSuspendStateSig)(BOOL, BOOL, BOOL);

which solved the problem. If you take a look in the header file where SetSuspendState is found (powrprof.h, part of the SDK), you will see the function prototype is defined as:

BOOLEAN WINAPI SetSuspendState(BOOLEAN, BOOLEAN, BOOLEAN);

So you guys are having a similar problem. When you are calling a given function from a .dll, its signature is probably off. (In my case it was the missing WINAPI keyword).

Hope that helps any future people! :-)

Cheers.

Silencing the check is not the right solution. You have to figure out what is messed up with your calling conventions.

There are quite a few ways to change the calling convetion of a function without explicitly specifying it. extern "C" will do it, STDMETHODIMP/IFACEMETHODIMP will also do it, other macros might do it as well.

I believe if run your program under WinDBG (http://www.microsoft.com/whdc/devtools/debugging/default.mspx), the runtime should break at the point where you hit that problem. You can look at the call stack and figure out which function has the problem and then look at its definition and the declaration that the caller uses.