C# DllImport with C++ boolean function not returning correctly
I found the solution for your problem. Your declaration should be preceded with this marshaling:
[return:MarshalAs(UnmanagedType.I1)]
so everything should look like this:
[DllImport("Whisper.dll", EntryPoint="Exist", CallingConvention=CallingConvention.Cdecl)]
[return:MarshalAs(UnmanagedType.I1)]
public static extern bool Exist([MarshalAs(UnmanagedType.LPStr)] string name);
I tested it in my very simple example and it worked!
EDIT
Why this happens? C defines bool as 4 bytes int (as some of you have said) and C++ defines it as 1 byte. C# team decided to use 4 byte bool as default during PInvoke because most of the system API function use 4 bytes values as bool. If you want to change this behavior you have to do it with marshaling specifying that you want to use 1 byte value.
C'sbool
is actually int
, as there is no boolean type in the original C language. That means that if C#'s DLLImport is designed to interop with C code, then they will expect that C#'s bool
to correspond to C's int
. While this still doesn't explain why false would become true, fixing it should fix the problem.
http://msdn.microsoft.com/en-us/library/system.runtime.interopservices.unmanagedtype.aspx
This says that UnmanagedType.Bool is the Win32 BOOL
, which is an int
.
This is actually caused by EAX not being fully cleared out by typical C++ code that returns a bool
. It's typical for EAX to contain some bogus value when entering a function, and to return false
the compiler would typically emit xor al, al
. This clears out only the LSB of EAX, and causes C# code to interpret the resulting non-zero value as true
instead of false
.
Perhaps marshaling the argument of the function might help:
[MarshalAs(UnmanagedType.LPStr)]
Here is how the declaration should look like:
[DllImport("Whisper.dll", EntryPoint="Exist", CallingConvention=CallingConvention.Cdecl)] public static extern bool Exist([MarshalAs(UnmanagedType.LPStr)] string name);