Can't pickle defaultdict

I have a defaultdict that looks like this:

dict1 = defaultdict(lambda: defaultdict(int))

The problem is, I can't pickle it using cPickle. One of the solution that I found here is to use module-level function instead of a lambda. My question is, what is module-level function? How can I use the dictionary with cPickle?

In addition to Martijn's explanation:

A module-level function is a function which is defined at module level, that means it is not an instance method of a class, it's not nested within another function, and it is a "real" function with a name, not a lambda function.

So, to pickle your defaultdict, create it with module-level function instead of a lambda function:

def dd():
    return defaultdict(int)

dict1 = defaultdict(dd) # dd is a module-level function

than you can pickle it

tmp = pickle.dumps(dict1) # no exception
new = pickle.loads(tmp)

Pickle wants to store all the instance attributes, and defaultdict instances store a reference to the default callable. Pickle recurses over each instance attribute.

Pickle cannot handle lambdas; pickle only ever handles data, not code, and lambdas contain code. Functions can be pickled, but just like class definitions only if the function can be imported. A function defined at the module level can be imported. Pickle just stores a string in that case, the full 'path' of the function to be imported and referenced when unpickling again.

You can however use partial to accomplish this:

>>> from collections import defaultdict
>>> from functools import partial
>>> pickle.loads(pickle.dumps(defaultdict(partial(defaultdict, int))))
defaultdict(<functools.partial object at 0x94dd16c>, {})