Interrupt installation when custom action returns error

I have a method that verifies a password in a .dll and it should return an error code on failure. It has the following prototype:

#define DllExport __declspec( dllexport )
extern "C" DllExport UINT TestPassword(MSIHANDLE hInstall);

I expect that when this method returns an error code (like ERROR_INSTALL_USEREXIT = 1602) the whole installation process will terminate (no other custom action following this one will be executed), but it doesn't.

Also, in Wix I have the following fragment:

 <CustomAction Id='TestPassword' BinaryKey='TestPassword' DllEntry='TestPassword' Execute='immediate'
              Return='check'/>

<Binary Id='TestPassword' SourceFile='DummyDll.dll'/>

Solution 1:

Quick Link: Managed Code Custom Actions (below is for C++ native custom actions).

UPDATE: Some helpful links.

• Hello WiX (minimal WiX Visual Studio Project). WiX quick start suggestions.
• Debugging Custom Actions (for native code / C++ just attach debugger to msiexec.exe).
• Microsoft Debugging Environments.
• Dynamic Link Libraries.

Suspected causes:

Listing some suggestions at the top here.

• 1) Wrong C++ custom action code configuration, often forgetting to create a CA.def file to define dll exports. I also always use __stdcall (MSI is an old girl).
• 2) Wrong path to custom action dll in WiX markup (not valid dll in MSI).
• 3) Forgot to enable checking of error codes in the WiX markup (*.WXS) and then a CA does not end the setup. This looks correct in your case (Return='check').
• 4) Forgot to insert custom action in installation sequence.
• There are a few more, can't think of them at the moment. Might add later... One that comes to mind is bitness problems (x86/64)...
• File and runtime dependencies is a classic fallover cause.
• Try to statically link whatever you can.
• Deployment DLLs should be minimum dependencies for sure since they need to run on any system, in any language, in any state, in any OS version, et...
• One of the few cases where static linking is really recommended and always the right choice.

Heads-Up: Avoid Licensing In Setup? I would recommend you put the license validation in your application instead of your setup. Here are some thoughts on the matter: Installer with Online Registration for Windows Application (recommended read).

Technical Issues:

FileName.def: I am no C++ expert, but do you have a FileName.def file in your project to declare the exported functions for the dll? If not - add one (steps / procedure below). Make sure it is in the right format (add via Visual Studio, I think it is UTF8 without BOM). Compile and check with Dependency Walker if all exports are correct:

Verify MSI File DLL: You should check the compiled MSI to verify it has the correct DLL inside it with the correct exports available. Hence; verify that the DLL has safely made it into the Binary table of the MSI:

1. Open your compiled MSI with Orca (or equivalent).
2. Binary table, double click the Data column for your DLL entry.
3. Select "Write binary to filename" and save to desktop (or somewhere else).
4. Use Dependency Walker (depends.exe) to verify that you have a valid DLL as illustrated in the image above. Common problem is that you see no exports at all (MyImmediateCA, MyTestFail, MyTestSuccess, etc...).
5. Verify the file- and product versions as well in file properties.

Error Processing: A custom action can be set to suppress errors. Your markup looks correct with the "Return attribute" set: (Return='check'). Your snippet:

<CustomAction Id='TestPassword' BinaryKey='TestPassword' 
              DllEntry='TestPassword' Execute='immediate' Return='check'/>

Sequencing: Also check that your sequencing is OK. Altogether you need to point to the binary table DLL, declare the custom action and then also insert it into the right sequence. Mock-up WiX markup:

<!--<Binary Id="CustomActions" SourceFile="$(var.TestDll.TargetPath)" />-->
<Binary Id="CustomActions" SourceFile="C:\TestDll.dll" />

<CustomAction Id="MyTestFail" BinaryKey="CustomActions" DllEntry="MyTestFail"/>
<CustomAction Id="MyTestSuccess" BinaryKey="CustomActions" DllEntry="MyTestSuccess"/>

<InstallExecuteSequence>
  <Custom Action="MyTestSuccess" After="CostFinalize" />
  <Custom Action="MyTestFail" After="MyTestSuccess" />
</InstallExecuteSequence>

C++ DLL: And the actual C++ DLL itself (remember the *.def file). Snippet in the bottom code segment from MSI API Custom Action Security:

Suggested steps for Visual Studio 2017:

1. Create new VC+ DLL Project - Dynamic-Link Library (DLL).
2. Dump the below code in the main *.cpp file (I avoid the dllmain.cpp).
3. Add the *.def file!

• Right Click Source Files => Add => New Item... => Code => Module-Definition File (.def) => Any name should do... (only one def file allowed)
• Add your export function names:

Mock-up:

LIBRARY

EXPORTS
     MyTestFail
     MyTestSuccess
     MyImmediateCA

Close and re-open file to verify if there are any format errors. Select fix if a warning appears. UTF8 without BOM required I think.

#include "stdafx.h"

#include <windows.h>
#include <Msiquery.h>
#pragma comment(lib, "msi.lib")

UINT __stdcall MyTestFail(MSIHANDLE hInstall)
{
    MessageBox(NULL, L"MyTestFail", L"MyTestFail", MB_OK);    
    return ERROR_INSTALL_FAILURE;
}

UINT __stdcall MyTestSuccess(MSIHANDLE hInstall)
{
    MessageBox(NULL, L"MyTestSuccess", L"MyTestSuccess", MB_OK);    
    return ERROR_SUCCESS;
}

// I will leave in the below snippet from the MSI API - section "Custom Action Security". Above two test methods will do though... 
UINT __stdcall MyImmediateCA(MSIHANDLE hInstall)
{
    MessageBox(NULL, L"Test", L"Test", MB_OK);

    // set up information for deferred custom action called MyDeferredCA
    const TCHAR szValue[] = TEXT("data");
    UINT uiStat = ERROR_INSTALL_FAILURE;
    if (ERROR_SUCCESS == MsiSetProperty(hInstall, TEXT("MyDeferredCA"), szValue))
    {
        uiStat = MsiDoAction(hInstall, TEXT("MyDeferredCA"));

        // clear CustomActionData property
        if (ERROR_SUCCESS != MsiSetProperty(hInstall, TEXT("MyDeferredCA"), TEXT("")))
            return ERROR_INSTALL_FAILURE;
    }

    return (uiStat == ERROR_SUCCESS) ? uiStat : ERROR_INSTALL_FAILURE;
}

Another answer on MsiGetProperty (retrieving property values). This is a little more complicated in C++ - with the buffers and all. Scroll down for source code.

Minimal Dependencies: In order to minimize dependencies you should eliminate the Visual C / C++ Runtime dependencies and any MFC dependencies (don't use MFC if you can help it for file size and performance reasons). If you use MFC, set it to use static linking - also for ATL. And finally for the C/C++ runtime, see here: Visual Studio 2010 MSVCR dependency removal? (there are better links, but all I could find that I have time for right now - just want to get this in there so it is not forgotten).

The Release mode C++ binary should now not depend on any MSVC runtime dlls:

Here is a screenshot of how the DLL depends on MSVC runtime dlls without this tweak - don't mind the red icons - this is the ancient dependency walker tool which is not updated for modern dependencies, but shows older-style dependencies perfectly:

Please note that debug-mode DLLs may depend on different files than the Release mode binaries. The Release mode binaries are the ones that are important. Obviously never distribute debug-mode binaries!