Why is it a good idea to limit the use of custom actions in my WiX / MSI setups?

Deployment is a crucial part of most development. Please give this content a chance. It is my firm belief that software quality can be dramatically improved by small changes in application design to make deployment more logical and more reliable - that is what this "answer" is all about - software development.

This is a Q/A-style question split from an answer that became too long: How do I avoid common design flaws in my WiX / MSI deployment solution?.

Solution 1:

Core: Essentially custom actions are complex to get right because of complexity arising from:

Sequencing-, Conditioning- (when it runs - in what installation mode: install, repair, modify, uninstall, etc...) and Impersonation-issues (security context it runs in).

It results in overall poor debugability - very hard hard work to test in all permutations.

Application Launch: Solution? Prefer application launch code. You get a familiar debugging context and can usually keep the code in the main application source - very important. The issue is the same for: licensing in setups (take it out if you can).

Custom Action Debugging: With all warnings in (you will read the short "folksy" version below - yes you will - Jedi trick) - do use built-in constructs when available - here is how you can improve debugging custom actions though:

Common Causes for Custom Action runtime failures

Debugging Custom Actions

For native code / C++ just attach debugger to msiexec.exe

Advanced Installer's Debug C# Custom Actions video tutorial

Short "Folksy" Version:

Overall: a modern setup should be "declared", not coded. "think SQL queries, not scripting". Apply well-tested logic that you just invoke to do the work. More on that later.

Avoid custom actions if you can.
- They will screw with your head (and do murder to your girly figure)!
- Seriously: they are the number one cause of deployment failure.
- They feature a "conspiratory complexity" whereby things often break unexpectedly and without warning after you have deployed your setup for some time and it is "in the wild".
- Deployment is a process - you will have to deal with "past sins" for every new release. It can become a real nightmare to clean things up after broken deployments. Each iteration adds new, potential error sources (my fix for my failed fix, failed, etc...).
- Runtime requirements for managed code and scripting custom actions are very error prone (.NET in particular in my opinion). A setup - if anything - should feature "minimum dependencies" - since it is used to install dependencies outright and should run on "any machine" in "any state" in "any language".
- Apart from using built-in MSI constructs or WiX custom features, custom actions can often be avoided by minor changes to the application design so that complex custom actions are no longer needed (samples below).
Spend the time to find and review built-in MSI features and extensions from WiX, Installshield, Advanced Installer (and other vendor tools).
- These extensions are made by the best deployment experts available, and have in most cases been tested by thousands of people (millions and billions in the case of built-in MSI features).
- How can you conceivably match this with your own proprietary code? If the existing solution is good enough - which it normally is - then your own code is entirely added risk for no gain whatsoever. The ready-made solutions even support rollback - a notoriously neglected and hard-to-implement feature - that is actually required by design.
- In other words: when you use the existing solutions, you borrow not just the implementation, but crucially also the QA, UAT and testing of the extensions when you use them. This can save weeks of work and contribute greatly to deployment stability.
- All that is needed is to spend some time searching and reading up on what is already available, rather than jump-starting things with your own code. Heck, even your own manager could get involved searching... In a dream world.
- A setup should generally not be coded, it should be declared! (think SQL). This is what Windows Installer is all about: you declare what to be installed, and the sequence is taken care of by the installer engine itself. Benefits result - when done right.
That's about it really. Read on if you dare! Here be dragons in the midst of serious ranting. And crucially: real-life experience from dealing with both setup development, and also large scale deployment of such setups in corporate environments. Common problems become clearer - they are more identifiable. And you may get burned with problems you really have to answer for - that you feel you could never have foreseen (doing corporate packaging and repackaging you get to see just how much can be wrong with a single package - it can take days to beat them into shape at times).
Please remember, and this is not meant as bad as it sounds: deployment is generally a low status technology endeavor, but a high profile failure. Errors really show. And someone will get to answer for it, and support will really feel it.
Debugging during development is hard, but for deployment it is sometimes nearly impossible since you generally have no access to the problem systems, and each problem system will be in a totally different state.
In all honesty: failing to get software deployed correctly to end users may be close to being the most expensive error to make in software development - nobody can witness the greatness of your solution - and it affects sales, support, marketing and further solution development.
Poor deployment can definitely sink a product. Unfortunately good deployment isn't enough to save a product.

Fleshed Out Ranting Version :-)

As stated above this section was split from an existing answer with broader scope: How do I avoid common design flaws in my WiX / MSI deployment solution? (an answer intended to help developers make better deployment decisions).

Erroneous or unnecessary use of custom actions.

A plethora of deployment problems can be caused by custom action use - most of them very serious. If your setup fails to complete or crashes, it is a fair bet that a erroneous custom action is at fault.

Accordingly the obvious solution is to limit your use of custom actions whenever possible. Custom actions are (often) "black box" (hidden code) whereas most of MSI features a lot of transparency. The MSI format can be viewed easily (COM-structured storage file), and everything that an MSI file does can essentially be deduced from its MSI database file - with the exception of compiled custom actions (script custom actions are still white box - you can see what is going on, unless they are obfuscated).

In this age of malware, these black box custom actions - that run with elevated rights - may become frowned upon in more ways than one. They are not just stability and reliability issues, but a full-blown security issue.

6.1 The Overuse of Custom Actions

Pardon the "trifles" listed below. A lot of this may be banality, but maybe use the arguments listed to make a case for yourself to avoid custom actions and to work on better solutions - usually involving smarter application launch sequence and application self-configuration. Trust us, application coding will be more interesting than dealing with Windows's Installer's complex conditioning, sequencing and impersonation for custom actions.
The below has been updated so many times that sections are a little bit out of sequence or repetitive here and there. Will be cleaned up as time allows.
Developers are good at coding - so with all due respect - you tend to overuse custom actions to do things that are better done using built-in MSI features or ready-made MSI extension solutions such as those available in WiX for advanced things such as XML file updates, IIS, COM+, firewall rules, driver installation, custom permissioning for disk and registry entries, modify NT privileges, etc... Such support is also found in commercial tools such as Installshield and Advanced Installer.
It is also possible to do a lot of what is done in custom action as part of an application's launch sequence. The prime example is initializing user data and copying settings file to each user profile. There is a little summary of this issue here: Create folder and file on Current user profile, from Admin Profile.
It is obvious, but very often custom actions are used out of ignorance of what is already available "out of the box" by means of ready-made solutions. This happens to all of us all the time, doesn't it? There is always some smarter way to do things that would have saved you lots of grief? In general anyway - though sometimes you are breaking new ground. Don't break new ground in your setup - unless you absolutely have to! Save it for your application.
I want to emphasize that these built-in Windows Installer solutions and extensions from WiX and commercial tools have been written by the best deployment specialists available. And moreover, and even more importantly, they have been used and tested by thousands, millions - heck even billions of people for built-in MSI features. Do you really think you can do any better? Moral of the story: pick your battles and use your great coding skills to solve new problems, and let deployment be as dumb as possible. Use what already works, and don't reinvent the wheel. There are too many unknowns in deployment, too many variables that can't be controlled - you are dealing with "any machine" in "any state" and in "any language". See the Complexity of Deployment section here if you want examples - a bit down the page - just a short dump of all the ways your target systems may differ in their states when your package hits it. Each variable is a new bear trap for custom code - from OS version, to application estate to malware situation. The list goes on and on. As the conclusion reads in the linked content: "Deployment is a simple concept, with a complicated mix of variables that can cause the most mysterious errors - including the developer favorite: the intermittent bug. As we all know the seriousness of such bugs can not be overstated as they are often impossible to debug properly."
Certain advanced things must be done in your setup - since they require elevated rights and your application should not request this whilst running. This is what a setup is for, advanced, elevated system configuration - so embrace this complexity, but use ready made solutions! Don't roll your own script and solutions, use built-in, well tested stuff. Will your script run correctly in Korea? Will your custom action be blocked by a major anti-malware solution of caliber that you never had the time to test your script with? Do you have time to write your own check for whether a certain prerequisite runtime is installed on the computer - which works in all locales and across all OS versions? The potential for bugs here is staggering - and sometimes impossible to test properly. Do you have test machines in Arabic, Chinese, Korean or Japanese? Maybe you do. But do you have a terminal server to test on? How about a Korean terminal server? Did you test your setup with MSI advertisement? For advanced system management features to work, you have to make your setup as dumb and standard as possible. Spend several days looking for ready-made solutions before you write anything on your own I'd say. Remember, with ready-made solutions you are not just borrowing their code, but crucially their creator's QA, testing and UAT - which you can almost never hope to repeat.
Real-world testing is the only yardstick that matters - there is no substitute. Don't take on this world of pain! A slight change in Windows deployed via a Windows Update and your custom action breaks in any number of ways. Choose a better battle to make use of your skills. If you fight the design, then Windows installer fights back.
And there is a lot going on with deployment that makes it complex - and not dumb like we want it (just copy some darn files), your fight is to make it as simple as possible, but no simpler. Here is a list of deployment tasks showing why it is hard to get your package "dumb enough": What is the benefit and real purpose of program installation? Use only ready-made constructs whenever you can - it is the first easy win. All you need to do is to read and search a little.
Finally I will add that all custom actions are supposed to support rollback to put the system back to the original state if the install fails. In the real-world this is almost never done in an ad-hoc custom action (in my experience). Believe me it is complicated to deal with - I fear the word "rollback" as a Siberian husky fears the word "bath" after I implemented MSI rollback in C++. Not the most fun I ever had, but it eventually worked properly. If you want your setup to be without too many dependencies, C++ is the way to go, and as we all know it is not trivial to deal with. The ready-made solutions support rollback out of the box - it is an easy-win for only a bit of reading and searching to enable them. Complexity yes, but you are on more solid ground. And crucially: once an obscure error happens on a Japanese machine we can work for a community fix, or a third party vendor needs to sort it.
After all those "general observations", on a purely technical level, custom actions in Windows Installer are very complex with regards to implementation, scheduling, conditioning and rollback, and should hence be used when absolutely necessary (often for early-adopter stuff). A further complexity is runtime requirements - for example dependencies on specific versions of the .NET runtime, or PowerShell, or Installscript runtimes (scripting language of Installshield - it had a runtime dependency, but this is largely solved now, it used to be a problem).
Errors from missing runtime requirements can be a enormous hairball to deal with - for zero gain. For this reason I only use minimal dependency C++ dll's or Installscript when making custom actions. It happens that I use VBScript or JavaScript for read-only custom actions in the user interface sequence. These just retrieve data, and make no system changes. These are the only types of custom actions that are not that error prone. They should however be set to run without checking exit codes (to prevent them from triggering rollback or abort in a setup that is being run - often in major upgrade mode).
Also: Windows Installer hosts its own scripting runtime engine, so you know your active scripts (VBScript, Javascript, etc...) can run unless your target system is actually broken (there is no missing runtime on a normal system). This is in contrast to managed code custom actions - it is entirely possible for target systems to have no .NET runtime at this point (Jan 2018). Now this will change in the future, as .NET becomes a really built-in and required feature in Windows (or some minimal version of it hosted by Windows Installer itself). There are still problems with managed custom action code failing for strange reasons - such as the wrong .NET version being used to run it, or the wrong version of the CLR being loaded and being used, etc... I have limited experience here, but the problems are serious in my opinion. Eventually, though, we will all write managed code custom actions I think. I would still use C++ for worldwide distribution scenarios though.
Powershell scripts are particularly hairy for custom actions as they apparently run out of process and can't access the MSI's session object (source: MSI expert Chris Painter). Powershell also requires the .NET framework to be installed. I would never use Powershell script for custom actions - not even for internal, corporate deployment. Your call. Just as a "sample", here is an expert in the field stating his opinion (aging blog item, but still relevant if you ask me): Don’t use managed code to write your custom actions!
I tried to write a summary of pros and cons of different custom action types. Frankly I am not too happy with it, but here it is: Windows Installer fails on Win 10 but not Win 7 using WIX. What I am not happy with? The recommendation of JavaScript for one thing - I have used it seldomly, and although it is a better language than VBScript - particularly for error handling - I have had lots of problems using Javascript with MSI. It may be a case of the problem existing between the keyboard and the chair :-), but I think there are some real gotchas with JavaScript as well. Try to avoid complex use of scripts. For simple things like retrieving properties they seem to work OK for me. However, the experts in the field are merciless on script custom actions: Don’t use vbscript/jscript to write your custom actions! (Aaron Stebner), VBScript (and Jscript) MSI CustomActions suck (Rob Mensching - WiX benevolency).
Let me also add that custom action complexity is "silly, conspiratory complexity" not fun-to-deal-with stuff. It bites you. Gotchas. You will discover all the mistakes you have made in due course - as you move along (and then you have some explaining to do) - they will often not be immediately obvious (problems suddenly arise when upgrading, patching, uninstalling, your custom action runs erroneously during repair because it is not conditioned properly and wipes out certain registry settings, silent install doesn't work properly - it leaves the install incomplete because the custom action only exists in the user interface sequence, there are runtime errors on Windows XP that you never tested your custom action code with, there are broken things on target systems that you didn't shield yourself from, someone calls you and tells you your package doesn't work with active directory, your package can't be advertised, it fails on all Korean and Japanese systems, self-repair triggers runtime warnings and access denied for normal users, etc...). You will have no time to fix this properly once it hits you, and you will likely have to proceed with sub-standard solutions to get yourself ahead over the next hurdle. And no, I didn't experience all of this myself. In fact I discovered most of these problems when fixing up third party vendor MSI files for corporate deployment. You really discover a lot of potential error sources when you see, compare, review and adapt hundreds of packages to a corporate standard for large scale deployment. In all honesty there are serious shortcomings and errors in all but the simplest of packages. And obviously you get to see what you did wrongly when you made such vendor setups a few years earlier. And guess what tops the list? Overuse of custom actions when built-in constructs were available.
And to add to it all, the inherent complexity of deployment with its requirement to work on any machine, anywhere in any state, there is the issue of MSI technology borderline anti-patterns. Parts of the MSI technology that are causing repeated deployment problems because they are poorly understood, and sometimes fragile in real-world use. There is a brief summary of this in this answer (towards the bottom): How to make better use of MSI files (along with a list of the very important corporate benefits of MSI - and a random dump of common technical problems seen in real-world packages). Particularly the latter link has struck me as less than ideal after I wrote it. Take it for what it is: messy, real-world advice without much else going for it. It identifies too many problems without showing much in the line of fixes in places.
A substantial proportion of a software's support calls tend to come from problems seen during the deployment process. As the story goes: "...failing to properly install your great software may be close to being the most expensive error to make in software development. You can never hope to sell a software that was never possible to test" (from one of the linked answers above). A bad custom action is very often behind such problems.
The most common unnecessary custom action uses we see are in my opinion:
- You install Windows Services via custom actions. This is much better done in the MSI itself using built-in constructs.
- You install .NET assemblies to the GAC via a custom action. This is fully supported by Windows Installer itself without a line of (risky) code.
- You run custom .NET assembly installer classes. These are to be used for development and testing only. They should never be run as part of deployment. Rather your MSI should use built-in constructs to deploy and register your assembly.
- You run prerequisite setups and runtime installers via a custom action in your own MSI. This should be done entirely differently. What you need is a bootstrapper that can launch your setup and its prerequisites in a sequence. Commercial deployment tools such as Advanced Installer and Installshield have features for this, but free frameworks such as WiX has support via its Burn feature, and there are also free GUI applications such as DOTNetInstaller (untested by me) with these kind of features.
- Please take my word for it in this particular case: running embedded setups via a custom action will fail eventually - usually rather immediately. MSI is too complicated in its scheduling, impersonation, transaction, rollback and overall runtime architecture to make this possible. Only one MSI transaction can run at a time by design, and this makes things very complicated. It used to be you could run embedded MSI files as a concept in MSI, but this was deprecated - it didn't work properly. You must run each setup in sequence. The correct sequence. There are bootstrappers / chainers available that will allow you to define such an installation sequence. Here is an answer which describes some of them (don't let the question title throw you off - it is about bootstrappers / chainers and other deployment considerations): Wix - How to run/install application without UI.

6.2 Custom Action Alternatives

The ready-made solutions found in WiX and other tools such as Installshield and Advanced Installer, are tried and tested, and crucially also implement proper rollback support - a feature almost always missing from custom action implementations - even in otherwise good vendor MSI setups. Rollback is a very important MSI feature. You cannot compete with the quality delivered from a large user community with active use and testing in all kinds of environments. Make use of these features.
Apart from using built-in MSI constructs or WiX custom features, custom actions can often be avoided by minor changes to the application design so that complex custom actions are no longer needed. I see licensing as an example of how deployment can be simplified to improve reliability.
- This is a very important point, and one that is very often ignored. Sometimes some quality hours or days of coding and some quality UAT / QA time can prevent long term deployment problems from ever existing. No amount of support time can solve the worst deployment problems you can create.
- This answer provides a blurb about the overall complexity of deployment: Windows Installer and the creation of WiX (towards the bottom). Deployment is a complicated "delivery process" with several serious challenges: 1) the cumulative nature of deployment errors, 2) the difficulty of proper debugging, and 3) the virtually unlimited range of external factors and variables affecting target machines world-wide - the conclusion is that deployment must be made as simple as possible to be reliable. There are so many intervening factors leaving you with the developer favorite: the intermittent bug. The link above is recommended for a more fleshed out explanation.

6.3 Advanced Custom Action Problems (beyond runtime failures)

A very common custom action issue is their incorrect scheduling and attempting to modify the system from "non elevated" immediate mode actions. These are advanced MSI design flaws not immediately recognizable for people who see an otherwise working setup.
Never insert immediate mode custom actions after InstallFinalize in the InstallExecuteSequence of your MSI that attempt to modify the system (read/write). These actions will never run if your setup is deployed in silent mode via deployment systems such as SCCM (again, last time I checked).
Never try to change the system using immediate mode custom actions activated from the setup GUI. This will appear to work for admin users, but immediate mode custom actions are not elevated and hence regular users will not be able to run them without error. In addition any changes to the system done from the MSI GUI will never be done when the setup is run in silent mode (then the whole GUI sequence is skipped). This is a very serious MSI design flaw: installing silently and interactively causes different results. This issue is mentioned in section 10 on silent install as well in this answer: How do I avoid common design flaws in my WiX / MSI deployment solution?. Silent install is a crucial feature for corporate deployment of your software. In corporations with control of their deployment, ALL deployment is done silently. Your setup GUI is simply never seen. There are only a few exceptions I can think of, such as certain server deployments that may be done interactively, but all desktop and client software is deployed silently. Not supporting silent install properly is hence a horrendous deficiency and design flaw of your MSI setup. Please heed this advice. It is crucial for corporate acceptance of your software - I have recommended certain software to be thrown out from the standard system because its deployment is so dangerous for the system that we just don't want to deal with it. Virtualization and sandboxing (APP-V - virtual packages - or full on virtual machines) are big today precisely because of these issues (misbehaving applications and setups).
All custom actions that make changes to the system must be inserted in the "transacted section" of the installation sequence. This Windows Installer Transaction (think database transaction commit) runs between the standard actions InstallInitialize and InstallFinalize in the InstallExecuteSequence and runs with elevated rights. All changes to the system are to take place in this transaction - anything else is erroneous (but unfortunately quite common). All custom actions inserted here should have a corresponding rollback custom action implemented as well, and this should undo any changes done to the system in case the install fails and gets rolled back.

Solution 2:

The idea that custom actions should be limited comes from the fact that it is easy to write a poorly implemented custom action. A poorly implemented custom action is one that, in the event of installation failure, cannot roll back changes already made to the system (i.e. installing a windows service, for example).

That being said, if a custom action is written such that it has a complimentary rollback custom action in place, then I don't think its a problem.

The pattern I like to follow is that for every "custom action" that I think I might need (for example, FoobarCustomAction), its actually 3 consists custom actions:

FoobarCustomActionImmediate - This is an immediate custom action that prepares 2 payloads:
- First, a payload to pass to FoobarCustomActionDeferred, which dictates the system altering changes to make.
- Second, a payload to pass to FoobarCustomActionRollback, which dictates how to rollback the system altering changes.
FoobarCustomActionRollback - This is a rollback custom action that is scheduled before FoobarCustomActionDeferred in sequence, that uses the payload provided by FoobarCustomActionImmediate to rollback the changes made by FoobarCustomActionDeferred in the event of a critical failure.
FoobarCustomActionDeferred - This is a deferred custom action that makes all of the system altering changes to the system.

Like many other technologies, I think there is always a way to write poorly maintainable code in WiX. Take the example above, if this is a custom action that I want to use in many different installers, I should provide the developers a .wxs file containing a fragment that ensures all needed custom actions are referenced and sequenced correctly. If instead, the WiX code is simply copy pasted carelessly across many different installers, the chance of the custom action being implemented incorrectly increases. Its our job as developers to help consumers of our code fall into the pit of success, and I think WiX provides that through careful use of fragment and wixlibs.

But the developer of the installer has to care about such things, WiX does not enforce it!