Is it normal to use "yes" begin a negative answer? [closed]

According to On the syntax of yes and no in English (alt link: download PDF), English uses the polarity-based system, in contrast to languages like Japanese that use the truth-based system. (This is exactly what Janus Bahs Jacquet alludes to in their comment.)

The reply, in your case, does not get its polarity from the original statement. Instead, its polarity comes from the word "yes", (so it's not wrong to say it that way).

It is normal usage in English.

I know that in French 'si' is used instead of 'oui' for 'yes' when it contradicts a negative, but English doesn't make that distinction. [1]