How can I avoid inviting a whole party of mercenaries when attacking a fort?

Don't be seen while killing people

Simply put, your bounty increases every time you kill someone and someone else is there to witness it. Killing the witness is pointless, because your bounty has already been increased.

If you're playing a Fighter, make sure you only fight enemies individually. Alternatively, you can fight groups of enemies, but you should avoid dealing the killing blow. Once the group is weakened enough, hide somewhere, and then pick them off without being seen by others.

If you're playing a Hunter, make sure nobody can see where you are, even if they witness the kill.

In addition to being sneaky, sabotaging the brazier should be one of your higher priorities. When lit, your bounty increases substantially.

Find the alarm brazier in the fort. When you get close to it you are prompted to 'set trap'. Once the trap is set anyone trying to light the alarm causes an explosion and the brazier is disabled. With no alarms the mercenaries don't turn up (unless you already have a bounty).