The more... the more
"succeed" is a verb. Verbs are modified by adverbs, but "great" and "greater" are adjectives. To turn an adjective into an adverb, you can add the "-ly" suffix, hence "greatly". Since you can't also use the "-er" suffix to express degree ("greaterly" is not a word), you have to use the word "more" to modify it.
However, in my opinion, "the more greatly" sounds awkward. I would write
The more robert tried to please his mother with mere flattery, the more he succeeded in annoying her.