Statistics Primer for the Unwary Mathematician

Adding to the other answers, look at: https://mathoverflow.net/questions/31655/statistics-for-mathematicians

And this one: https://stats.stackexchange.com/questions/63760/inference-for-the-skeptical-but-not-math-averse-reader (especially my answer, which I still think is good!)

I might suggest Parts II and III Wasserman's "All of Statistics." Wasserman's a good mathematician and great statistician, although the book is not aimed at people with an unusually good mathematical background-just at those wanting to use statistics well and quickly. In particular, "well" implies that it's rigorous. The book is essentially frequentist. There's a followup, IIRC, on non-parametric stats.