Reference request in number theory for an analyst.

Have a look at Koblitz: Introduction to Elliptic Curves and Modular Forms, and also at Knapp: Elliptic Curves. I prefer the latter, since it is more in depth, but it is also more algebro-geometric. The former has a much stronger complex analysis slant, especially at the beginning, which might make the entry easier. So you could try reading Koblitz first, and then Knapp (there will be a lot of overlap, of course).

The most thorough text on elliptic curves is Silverman: Arithmetic of Elliptic Curves. But it is considerably more algebro-geometric than the above two, and it has very little material on modular forms. So I am not sure it is the right entry text for you.

None of these cover complex multiplication. For that, you could have a look at Silverman: Advanced Topics in the Arithmetic of Elliptic Curves. My feeling is that to appreciate the theory of complex multiplication, it would help to have seen class field theory beforehand. But Silverman does review the main results of class field theory, so perhaps you can dive straight in, after having worked through one of the basic texts above.

"Just for fun" is a great premise to start learning elliptic curves, since it really is great fun!

A really good but underrated book that is freely available online is Milne's Elliptic Curves. Its treatment of elliptic curves is certainly less detailed than Silverman's but it is fairly through and yet accessible. He also has notes on complex multiplication. Also Silverman and Tate's book could give you a cursory exposure to elliptic curves and assumes nothing but a knowledge of groups.

Now with your experience in complex analysis, maybe the study of modular forms and analytic number theory would be of interest to you? Apostol has a series of two books (Analytic Number Theory and Modular Functions and Dirichlet Series) that develop these in some detail and are very classic books. Ram Murty's book has a ton of exercises which is always good when learning.

A knowledge of analytic number theory and modular forms will certainly not go to waste if you are interested in elliptic curves and are very interesting subjects in themselves as well.

EDIT: Since Milne clicks well with you maybe you should take a look at his entire collection of notes and books? He has stuff on algebra so you can use those to brush up on it.