How to deal with the temporary nature of my knowledge?
In my opinion, you have answered your own question when you stated:
I also make it a point to see the "big picture" rather than get caught in details, especially with proofs.
Good mathematical intuition is the result of hands-on practice, not its substitute. Yes, you must also keep the big picture in mind, but if you really want to learn this stuff, you have to get your hands dirty. Especially with proofs.
(I also agree with @skullpatrol's answer.)
In my opinion, you have answered your own question when you stated:
" I am unable to concentrate of my math for some time, when I return back a few weeks later (2-3 weeks at most) I seem to have lost it all. "
Simply don't not concentrate on math for that long. Do some sort of review daily.
Learning math is like learning anything else. It is most effectively done with daily time set aside for the task. Promise yourself 20 minutes per day on task.
Try to give yourself a set of mental exercises that you can do while involved in mundane activities. For example, I am trying to improve my arithmetic skills. In particular, i would like to be able to effortlessly multiply 2 digit numbers mentally. To do so, I need to create a database of particular products, know all the squares from 1-100, know all the products of successors x(x+1), and be able to subtract. When I swim, as I count my laps, I compute the squares and successors. When I walk, I do some sequential subtraction problems. I also do routine calculations when washing dishes, showering, and doing my toilet. Such methods depend on a short-term memory, and develop long-term memories. To be honest, I can't do the multiplication trick yet. I anticipate it will take a few more years to develop.
But if you are trying to learn calculus, then compute derivatives as you walk. If you are trying to learn linear algebra, then imagine intersections among planes as you shower. If you are trying to learn group theory, try listing the symmetries of a polygon as you go to sleep. Avoid visual mathematical calculations while driving!