Scrolling using Magic Mouse sometime not work if I don't use the mouse for a few minutes
This, I'm afraid, is the default behaviour for all modern battery-powered mice, not just Apple's.*
To save power they sleep after a very short period of inactivity & need to be woken again before they'll respond.
Empirically, I've noticed that mouse movement achieves this faster than any more advanced movement such as scrolling.
I see no workaround, other than abandoning wireless mice & returning to wired.
*Obviously, I haven't tested every mouse in the world but we have an assortment of Apple & Logitech devices here, all of which exhibit this behaviour.