What mechanic prevents Desolator and Mask of Madness from stacking?

This is because both are unique attack modifiers so they won't stack.

You are basically giving up their main purpose (either the armor reduction or life steal depending on which item is older).

You can still get the benefit of the passive and activation ability but those are not nearly as cost effective.