Solution 1:

The "Pronouncing Shakespeare's Words: A Guide from A to Zounds" appears to suggest that the pronunciation has something to do with on-stage usage during Shakespeare times:

  • Mentor, orator and other words of this class that are not commonly used in day-to-day life (including "Shakespearean" words like servitor, proditor, paritor) are often pronounced on- and off-stage with /-or/, but the traditional pronunciation is /-ur/, as found in most of our common words ending in -or (actor, instructor, doctor).

Garner's Modern English Usage refers to the pronunciation of mentor as "ˈmɛntər" and adds that:

  • the overpronounced "mɛnˌtɔr" is probably dominant in AmE today.

Solution 2:

Most likely, Mentor is pronounced differently from actor etc because it was derived from a name.

"wise adviser," 1750, from Greek Mentor, friend of Odysseus and adviser of Telemachus (but often actually Athene in disguise) in the "Odyssey," perhaps ultimately meaning "adviser," because the name appears to be an agent noun of mentos "intent, purpose, spirit, passion" from PIE *mon-eyo- (source also of Sanskrit man-tar- "one who thinks," Latin mon-i-tor "one who admonishes"), causative form of root *men- (1) "to think." The general use of the word probably is via later popular romances, in which Mentor played a larger part than he does in Homer. - etymonline

Here are a couple other Greek names that follow this pronunciation (links are to wikipedia):

  • Agenor - /əˈdʒiːnɔːr/; Greek: Ἀγήνωρ, Agēnor
  • Hector - Ἕκτωρ Hektōr, pronounced [héktɔːr]