Unload a module in Python

TL/DR:

import gc, sys

print len(gc.get_objects()) # 4073 objects in memory

# Attempt to unload the module

import httplib
del sys.modules["httplib"]
httplib = None

gc.collect()
print len(gc.get_objects()) # 6745 objects in memory

UPDATE I've contacted Python developers about this problem and indeed it's not going to be possible to unload a module completely "in next five years". (see the link)

Please accept that Python indeed does not support unloading modules for severe, fundamental, insurmountable, technical problems, in 2.x.

During my recent hunt for a memleak in my app, I've narrowed it down to modules, namely my inability to garbage collect an unloaded module. Using any method listed below to unload a module leaves thousands of objects in memory. In other words - I can't unload a module in Python...

The rest of the question is attempt to garbage collect a module somehow.

Let's try:

import gc
import sys

sm = sys.modules.copy()  # httplib, which we'll try to unload isn't yet 
                         # in sys.modules, so, this isn't the source of problem

print len(gc.get_objects()) # 4074 objects in memory

Let's save a copy of sys.modules to attempt to restore it later. So, this is a baseline 4074 objects. We should ideally return to this somehow.

Let's import a module:

import httplib
print len(gc.get_objects()) # 7063 objects in memory

We're up to 7K non-garbage objects. Let's try removing httplib from sys.modules.

sys.modules.pop('httplib')
gc.collect()
print len(gc.get_objects()) # 7063 objects in memory

Well, that didn't work. Hmm, but isn't there a reference in __main__? Oh, yeah:

del httplib
gc.collect()
print len(gc.get_objects()) # 6746 objects in memory

Hooray, down 300 objects. Still, no cigar, that's way more than 4000 original objects. Let's try restoring sys.modules from copy.

sys.modules = sm
gc.collect()
print len(gc.get_objects()) # 6746 objects in memory

Hmmm, well that was pointless, no change.. Maybe if we wipe out globals...

globals().clear()
import gc # we need this since gc was in globals() too
gc.collect()
print len(gc.get_objects()) # 6746 objects in memory

locals?

locals().clear()
import gc # we need this since gc was in globals() too
gc.collect()
print len(gc.get_objects()) # 6746 objects in memory

What the.. what if we imported a module inside of exec?

local_dict = {}
exec 'import httplib' in local_dict
del local_dict
gc.collect()
print len(gc.get_objects())  # back to 7063 objects in memory

Now, that's not fair, it imported it into __main__, why? It should have never left the local_dict... Argh! We back to fully imported httplib. Maybe if we replaced it with a dummy object?

from types import ModuleType
import sys
print len(gc.get_objects())  # 7064 objects in memory

Bloody.....!!

sys.modules['httplib'] = ModuleType('httplib')
print len(gc.get_objects())  # 7066 objects in memory

Die modules, die!!

import httplib
for attr in dir(httplib):
    setattr(httplib, attr, None)
gc.collect()
print len(gc.get_objects())  # 6749 objects in memory

Okay, after all attempts, the best is +2675 (nearly +50%) from starting point... That's just from one module... That doesn't even have anything big inside...

Ok, now seriously, where's my error? How do I unload a module and wipe out all of it's contents? Or is Python's modules one giant memory leak?

Full source in simpler to copy form: http://gist.github.com/450606

Solution 1:

Python does not support unloading modules.

However, unless your program loads an unlimited number of modules over time, that's not the source of your memory leak. Modules are normally loaded once at start up and that's it. Your memory leak most likely lies elsewhere.

In the unlikely case that your program really does load an unlimited number of modules over time, you should probably redesign your program. ;-)

Solution 2:

I'm not sure about Python, but in other languages, calling the equivalent of gc.collect() does not release unused memory - it will only release that memory if/when the memory is actually needed.

Otherwise, it makes sense for Python to keep the modules in memory for the time being, in case they need to be loaded again.

Solution 3:

Python's small object manager rarely returns memory back to the Operating System. From here and here. So, stricktly speaking, python has (by design) a kind of memory leak, even when objects are "gc collected".