Why does my Firefox memory usage keep rising with use and never returns to the initial level?

Enter about:memory in the address bar. Click 'Minimize memory usage'.

https://support.mozilla.org/en-US/kb/firefox-uses-too-much-memory-ram

The first thing I would suspect is that some of plug-ins leak memory or use it a lot.

For example, AdBlock Plus often causes Firefox to increase memory usage significantly. The reason is that it injects large style sheets into every frame. (Ghostery is a more memory-efficient alternative, although it serves a different purpose.)

So, as suggested in the comments, if memory usage doesn't seem reasonable, try disabling all plug-ins and see if this helps.