What do Disease Broodoo Victims deal damage with?
Each colored Broodoo Victim is weak to something.
The yellow ones are weak to Relicym's Balm that you mention. So it is just meant to kill these guys rather than protect you.
The white ones are weak to food easiest being Cooked Meat bought from the general store but MANY other options available. Basically any food item.
The green ones are weak to Antipoison.
Here is the wiki page that describes them all in full depth: Broodoo Victims