Mathematics and Physics prerequisites for mirror symmetry

First of all, Mirror Symmetry is huge. As you said, there are many fields involved. To know how much you need to know depends on where you're working. Roughly, one can divide the whole mathematical aspects of mirror symmetry into two categories. 1) Analytic and symplectic, (mainly (complex) differential geometry/symplectic geometry) 2) Algebraic (containing Algebraic geometry, homological algebra, etc.) I've been around with people who're doing Donaldson-Thomas theory (One Algebraic geometry side of Mirror symmetry) and personally willing to know more about homological mirror symmetry these days. Unfortunately, I don't know much about the analytic aspect which is related to Gromov-Witten theory.

The connections between these two categories are related to conjectures, one is called MNOP conjecture and the other interesting one is the homological mirror symmetry program.

As for the books and references, if you want to know just very little about what's going on, you may find Mirror symmetry written by leading mathematicians as well as mathematical physicist useful.However, Mirror symmetry and Algebraic geometry by Cox and Katz satisfies me more than the previous book (because obviously it's more mathematics.)