Reference request: books that describe application of physical reasoning to mathematical problems
Solution 1:
Let me mention a recent occurrence of a physical idea that played a key role in finding a proof for an abstract mathematical result. It was used in this paper. (My purpose is not to advertise the paper but to provide a concrete example. Physical intuition was very important in finding the theorem and the proof.)
The situation is somewhat technical, so don't worry if you don't get all of it. I will not try to describe the results of that paper, but only the bit where physics helped. Consider a Riemannian manifold with strictly convex boundary and a geodesic $\gamma$ that stays near the boundary and bounces when it hits it. (Such a curve is called a broken ray.) Let $\gamma^0(t)$ be the distance of $\gamma(t)$ to the boundary. It turns out that this "height from the boundary" obeys an equation of motion very similar to Newton's second law for a ball thrown in the air, and the bounces are like the bounces of a perfectly elastic ball. The difference is that the gravitation acceleration $g$ (or the geometrical thing corresponding to it, the second fundamental form) depends on position and direction. If one takes the Newtonian formula for the total energy $E$ of the ball, $E=g\gamma^0+\frac12(\dot\gamma^0)^2$ (unit mass), it is not exactly conserved (because gravitation varies), but it is almost conserved. That was useful.
Moreover, in the Newtonian case of a bouncing ball it is easy to verify that the average height of a ball is $\frac{2E}{3g}$. This is also the case the general setting, when interpreted properly. This idea suggested how to formulate and prove some lemmas that were crucial for the main result. This physical heuristic idea is described in equation (13) and the surrounding paragraph in the paper.