Breaking or break the habit
Solution 1:
The first, because the habit is presumably broken once and not continuously.
I would also rephrase slightly: "Imagining the ill effects of smoking ultimately led him to break the habit."
Out of interest, using a gerund like this also sounds okay: "Imagining the ill effects of smoking ultimately led to his breaking the habit."
But I would prefer the first.
Solution 2:
I think that it could be legitimately worded two ways depending on usage of "led".
You could use "led" in the transitive, more active sense. Per M-W: "to direct on a course or in a direction."
"(It) led him to quit ..."
You can also use "led" in the intransitive, more passive sense. Again per M-W: "to tend toward or have a result."
"(It) led to him quitting ..."
or alternately "(It) led to his quitting..."
Since the "It" in this sense is some abstract imagining, I think the intransitive use fits a little better, but either would be grammatically correct.