When is it better to build new towers rather than upgrade?

In general you will want to max out your Defense Units. This will actually lead to a pretty good award boosting your XP gains by quite a bit if you can keep the Defense Units maxed out for the entire round with out losing one.

Once that has been taken care of you can start to upgrade. Focus on key choke points and consider upgrading vs repair when appropriate (remember you will suck in any size mana gem around you for being even 1 point below max so that repair can end up being very costly). Also of note, is that while the damage bonus to upgraded turrets is nice you will often find yourself upgrading them more because of their durability. Do not get me wrong, the damage increase is very nice, but knight's defense having more health, the monk auras longer life and the huntress traps more trigger counts are also very key reasons to upgrade them, so they can last with out you being there.

Lastly, as yx. mentions is the blocking of enemies. Some turrets (ie knights) do this better than others (monks/huntress). Controlling the flow of the enemies is also often very key so the quantity of the towers can be more important than the damage they do as well.