Does Jinx increase the additional damage of Hex to 20% or 30%?
It should be 20%. Other runes, like Several Sided Strike (9 hits on Seven-Sided Strike), which are worded similarly do not mean original + rune (it's 9 hits, not 16), and Jinx should be no exception. It modifies the last sentence from "hexed targets... take 10% additional damage" to "hexed targets ... take 20% additional damage".
If it was 30%, the tooltip would most likely read, "Hexed targets take an additional 20% damage".