Mathematics needed in the study of Quantum Physics
Solution 1:
I am going to plug my undergraduate professor's book again, but it is honestly the best book I know to prepare oneself for the math involved in QM. (I should know, as I experienced his course as a Math/Physics double major.) The book is Applied Analysis by the Hilbert Space Method by Samuel S. Holland. It is now available in paperback and relatively inexpensive. This book is custom tailored for the math and physics student on the cusp of taking a first course in QM. There's even a chapter on the Schroedinger equation, with the solution to the hydrogen atom worked out in detail. I cannot recommend highly enough.
Solution 2:
It depends on what type of QM course you want to take. Courses in QM for engineers, undergraduate physics majors, graduate students in physics, and graduate students in mathematics are all pretty different. I will assume you're seeking an "undergraduate physics major" understanding of QM.
I have two recommendations:
In my opinion, the best Quantum Mechanics book for self-study is Shankar. My main reason is that the first third or so of the book is a survey of the mathematics you'll need in the other two thirds, i.e., it answers precisely the question you've asked. You'll need to know calculus first (including vector calculus), but from there Shankar will give you what you need. I also like this book because it includes tons of fully worked out examples, and pages upon pages of "what does this mean" type exposition. Usually, you'll see this book being used in graduate or advanced undergraduate quantum mech courses, but that is just because it is a very long book about a very involved subject, it doesn't mean it isn't accessible to the beginner.
When I took quantum mech I read Lang's Linear Algebra concurrently, and it made everything so, so much easier. You don't necessarily have to finish it, basically just get real comfortable with inner products and dual spaces, up through the spectral theorem.
Some additional remarks:
I think it's an exaggeration to say a course in quantum mechanics requires functional analysis or operator theory, even though that is essentially what you're doing. The mathematically rigorous forms of those disciplines are pretty advanced, but you don't need to understand them at that level to do quantum mechanics. You just need to be able to use them. Shankar will teach you how to do that. (Of course, I wouldn't discourage you from learning them eventually, but they aren't strictly necessary for the purposes of QM.) He should catch you up on the basics of probability theory, too.
I would not say that about abstract linear algebra, however. You will need to understand vector spaces, dual bases, inner products, eigenvalues, etc. on a rigorous level.
Group theory and representation theory are definitely not necessary. Although they are important to quantum mechanics, you will not see them until very advanced levels.
So, long story short: you need to know calculus up through vector calculus and linear algebra up through abstract linear algebra.
Addendum.
It's worth mentioning that, although what I've mentioned above would be sufficient for the mathematics side of things, it would be real good if you had seen mechanics, electricity and magnetism, and thermodynamics beyond an introductory level beforehand. It's possible to do it without previous physics courses, but you'll find yourself saying "so what?" a lot, as the weirdness of quantum mech will not seem as jarring to you if you have not seen how things work at larger scales.
Solution 3:
I think that Higher Maths for Beginners – Zeldovich, Yaglom and Elements of Applied Mathematics books have the math you need for QM. They're written by Zeldovich, a co-father of Soviet nuclear bomb project.
The latter book has a related volume called Elements of math physics. Noninteracting particles, unfortunately it's not translated to English. The Russian version used to be my favorite text for math physics.
One of my favorite math physics text was Methods of Theoretical Physics by Morse and Feshbach. It's huge, and is written like a handbook, no need to read the whole thing.
If you read German, then there's this crazy handbook The mathematical tools of a physicist by Erwin Madelung, I'm not sure if it's available in English, but I saw it in Russian.
Solution 4:
I have taken two courses in QM. There were a range of different math courses that are extremely useful. I would recommend studying multi-variable calculus, linear algebra, partial differential equations and probability theory.
Solution 5:
It depends a lot on the point of view you want to take on QM. For an experimental physicist's point of view, you will need real/complex analysis, linear algebra and probability theory. If you want the theoretical physicist's/mathematician's point of view, then add functional analysis (maybe with a focus on $C^*$-algebras/algebras of operators) and representation theory.