"Having not" vs "not having"

To me, having not given smells a bit like to not care, a split infinitive. Not having had much experience in this area, though, I'm interested to hear what other people think.

Both versions are fine, but not having given up does sound a little bit more fluent. If you want to express it in the affirmative tone, I would rephrase it as:

"Congratulations on not giving up yet!"

I would say that generally when using "having + VERB" you are expected to provide more detail because it is hinting that there is something else to follow. For example: "Having to deal with this problem is giving him more things to worry about."