Serious word for 'a bad guy'

I would change the wording to refer more to the attack than the person.

"Had it been malicious, your account would have been stolen."

Attacker fits the tone of rest of the message, and in prticular the use of the word attack. By adding real or actual you can differentiate yourself from the "bad guy" since you are technically using the vulnerability yourself.

Malicious user can be an alternative, since it directly describes the bad behavior.

Adversary is often used in infosec, but is typically used in a more hypothetical and technical tone and might not fit you warning message.

The site has an XSS vulnerability. An adversary could execute client-side code under conditions X and Y.

I think people understand hacker in this context.

Had it been made by a hacker, your account would have been stolen.

Antagonist or real attacker seem like they might work here.