Is it more correct to say "repeat", or "resay"?

First of all, "repeat" doesn't actually have a prefix. So, you aren't "peating" something again.
It's a word derived from French:

repeat
late 14c., from O.Fr. repeter "say or do again, get back, demand the return of" (13c.), from L. repetere "do or say again, attack again," from re- "again" + petere "go toward, seek, demand, attack" (see petition). Specific meaning "to take a course of education over again" is recorded from 1945, Amer.Eng. Related: Repeated; repeating. The noun is first recorded 1550s.

It has a prefix in Latin, but not in English.

As you can see, "repeat" can mean 'to do or say again', so "repeat" can be used to mean to 'resay'.

Interestingly, etymonline.com and dictionary.com both don't give records of 'resay'.

FWIW, in the Army, we were taught that if a radio transmission was unclear, then to say, "say again" ... NEVER repeat. Repeat has a specific meaning, it means to fire again. Thus if you're talking to an artillery unit and say "repeat" over the radio, they will fire again.

While resay does exist, it is better to say, "say again".

If skaldic muses are pushing you to write poetry, you could always use the seldseen (rare) prefixes of ed- (= re-) or agen- (= re-, again) ... agenbite = remorse.

However, for talking, use "say again".

If I didn't quite hear what you said (not loud enough, indistinct, etc.), I would ask you to repeat. If I heard every word you said, but I didn't understand what you meant, I would ask you to restate what you said; "resay" doesn't sound felicitous to me.

In English, of course, we generally just say "What?" in either case. In Spanish you could say "¿Qué?" ("What?") if you didn't hear, or "¿Cómo?" ("How?") if you didn't understand; in Russian it's the same: "Что?" or "Как?" I wish we had a similar convention in English conversation...