How can I pickle a dynamically created nested class in python?

Solution 1:

I know this is a very old question, but I have never explicitly seen a satisfactory solution to this question other than the obvious, and most likely correct, answer to re-structure your code.

Unfortunately, it is not always practical to do such a thing, in which case as a very last resort, it is possible to pickle instances of classes which are defined inside another class.

The python documentation for the __reduce__ function states that you can return

A callable object that will be called to create the initial version of the object. The next element of the tuple will provide arguments for this callable.

Therefore, all you need is an object which can return an instance of the appropriate class. This class must itself be picklable (hence, must live on the __main__ level), and could be as simple as:

class _NestedClassGetter(object):
    """
    When called with the containing class as the first argument, 
    and the name of the nested class as the second argument,
    returns an instance of the nested class.
    """
    def __call__(self, containing_class, class_name):
        nested_class = getattr(containing_class, class_name)
        # return an instance of a nested_class. Some more intelligence could be
        # applied for class construction if necessary.
        return nested_class()

All that is left therefore, is to return the appropriate arguments in a __reduce__ method on FloatType:

class WidgetType(object):

    class FloatType(object):
        def __reduce__(self):
            # return a class which can return this class when called with the 
            # appropriate tuple of arguments
            return (_NestedClassGetter(), (WidgetType, self.__class__.__name__, ))

The result is a class which is nested but instances can be pickled (further work is needed to dump/load the __state__ information, but this is relatively straightforward as per the __reduce__ documentation).

This same technique (with slight code modifications) can be applied for deeply nested classes.

A fully worked example:

import pickle


class ParentClass(object):

    class NestedClass(object):
        def __init__(self, var1):
            self.var1 = var1

        def __reduce__(self):
            state = self.__dict__.copy()
            return (_NestedClassGetter(), 
                    (ParentClass, self.__class__.__name__, ), 
                    state,
                    )


class _NestedClassGetter(object):
    """
    When called with the containing class as the first argument, 
    and the name of the nested class as the second argument,
    returns an instance of the nested class.
    """
    def __call__(self, containing_class, class_name):
        nested_class = getattr(containing_class, class_name)

        # make an instance of a simple object (this one will do), for which we can change the
        # __class__ later on.
        nested_instance = _NestedClassGetter()

        # set the class of the instance, the __init__ will never be called on the class
        # but the original state will be set later on by pickle.
        nested_instance.__class__ = nested_class
        return nested_instance



if __name__ == '__main__':

    orig = ParentClass.NestedClass(var1=['hello', 'world'])

    pickle.dump(orig, open('simple.pickle', 'w'))

    pickled = pickle.load(open('simple.pickle', 'r'))

    print type(pickled)
    print pickled.var1

My final note on this is to remember what the other answers have said:

If you are in a position to do so, consider re-factoring your code to avoid the nested classes in the first place.

Solution 2:

The pickle module is trying to get the TextType class from the module. But since the class is nested it doesn't work. jasonjs's suggestion will work. Here are the lines in pickle.py responsible for the error message:

    try:
        __import__(module)
        mod = sys.modules[module]
        klass = getattr(mod, name)
    except (ImportError, KeyError, AttributeError):
        raise PicklingError(
            "Can't pickle %r: it's not found as %s.%s" %
            (obj, module, name))

klass = getattr(mod, name) will not work in the nested class case of course. To demonstrate what is going on try to add these lines before pickling the instance:

import sys
setattr(sys.modules[__name__], 'TextType', WidgetType.TextType)

This code adds TextType as an attribute to the module. The pickling should work just fine. I don't advice you to use this hack though.

Solution 3:

If you use dill instead of pickle, it works.

>>> import dill
>>> 
>>> class WidgetType(object):
...   class FloatType(object):
...     pass
...   class TextType(object):
...     pass
... 
>>> class ObjectToPickle(object):
...   def __init__(self):
...     self.type = WidgetType.TextType
... 
>>> x = ObjectToPickle()
>>> 
>>> _x = dill.dumps(x)
>>> x_ = dill.loads(_x)
>>> x_
<__main__.ObjectToPickle object at 0x10b20a250>
>>> x_.type
<class '__main__.TextType'>

Get dill here: https://github.com/uqfoundation/dill

Solution 4:

In Sage (www.sagemath.org), we have many instances of this pickling issue. The way we decided to systematically solve it is to put the outer class inside a specific metaclass whose goal is to implement and hide the hack. Note that this automatically propagate through nested classes if there are several level of nesting.