In which order do hit effects apply?

Solution 1:

From my experimentation, all hit effects appear to be resolved before the game checks to see if the target is dead. As the damage effects resolve, if the corpse is already at 0 HP, they simply do nothing and it moves to calculating the next effect. Once all calculation is done, it checks to see if the target is still alive and if not, triggers the "on death" script actions.

This means that Soul Trap will still trigger and fill your gem even if the target was "overkilled" by a huge amount.

On a more general note, be very careful with enchanting Soul Trap into your standard combat weapons. If you don't keep a very large supply of soulgems of all sizes, it's very easy to accidentally run out of low-grade gems and start wasting your high-grade soulgems with low-grade souls. It's usually a better idea to have a dedicated Soul Trap weapon or just learn and use the spell.

Solution 2:

I 1-shotted an enemy with my Skyforge Steel Waraxe of Souls today (it was the special finishing animation and everything) and got its soul.

Judging from that, Soul Trap is pretty high priority, and should always be applied (assuming it isn't resisted / you have enough charge).