Should VCenter be a VM?/Can it be external hardware

There's no reason why not. I'm not aware if VMware specifically direct you to run vCenter on either physical or virtualised hardware, I believe it's supported on both.

Depending on what sorts of failures you're trying to protect against, it's usually a good idea to separate your redundant / standby instances from the primary / live instances as much as possible. Separate networks, cabs, power supplies and even buildings, cities and countries are all good ideas - they just cost different amounts and come with their own unique set of challenges.

In this particular case, it sounds like you had one of those outages which you hadn't designed / accounted for, or had knowingly chosen not to design around. Putting your management servers on the same infrastructure as the ESXi hosts, along with the same power supplies, networks etc. all runs the same risk of a single event taking everything out.

You have a choice - either you can choose to not change anything and live with the outages that result from this particular type of event happening again, or choose not to and spend some money to mitigate. Either is a valid approach, it entirely depends on how much it's going to cost you in outages vs how much it'll cost you to change.