Can Python pickle lambda functions?

I have read in a number of threads that Python pickle/cPickle cannot pickle lambda functions. However the following code works, using Python 2.7.6:

import cPickle as pickle

if __name__ == "__main__":
    s = pickle.dumps(lambda x, y: x+y)
    f = pickle.loads(s)
    assert f(3,4) == 7

So what is going on? Or, rather, what is the limit of pickling lambdas?

[EDIT] I think i know why this code runs. I forgot (sorry!) i am running stackless python, which has a form of micro-threads called tasklets executing a function. These tasklets can be halted, pickled, unpickled and continued, so i guess (asked on the stackless mailing list) that it also provides a way to pickle function bodies.

Yes, python can pickle lambda functions… but only if you have something that uses copy_reg to register how to pickle lambda functions -- the package dill loads the copy_reg you need into the pickle registry for you, when you import dill.

Python 2.7.8 (default, Jul 13 2014, 02:29:54) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import dill  # the code below will fail without this line
>>> 
>>> import pickle
>>> s = pickle.dumps(lambda x, y: x+y)
>>> f = pickle.loads(s)
>>> assert f(3,4) == 7
>>> f
<function <lambda> at 0x10aebdaa0>

get dill here: https://github.com/uqfoundation

Python can pickle lambdas. We will cover Python 2 and 3 separately as implementation of pickle are different in different Python versions.

Python 2.7

pickle uses pickle registry which is nothing but a mapping from type to the function to use for serializing (pickling) objects of that type. You can see pickle registry as:

>> pickle.Pickler.dispatch

{bool: <function pickle.save_bool>,
 instance: <function pickle.save_inst>,
 classobj: <function pickle.save_global>,
 float: <function pickle.save_float>,
 function: <function pickle.save_global>,
 int: <function pickle.save_int>,
 list: <function pickle.save_list>,
 long: <function pickle.save_long>,
 dict: <function pickle.save_dict>,
 builtin_function_or_method: <function pickle.save_global>,
 NoneType: <function pickle.save_none>,
 str: <function pickle.save_string>,
 tuple: <function pickle.save_tuple>,
 type: <function pickle.save_global>,
 unicode: <function pickle.save_unicode>}

To pickle custom types, Python provides copy_reg module to register our functions. You can read more about it here. By default, copy_regmodule supports pickling of the following additional types:

>> import copy_reg
>> copy_reg.dispatch_table

{code: <function ipykernel.codeutil.reduce_code>,
 complex: <function copy_reg.pickle_complex>,
 _sre.SRE_Pattern: <function re._pickle>,
 posix.statvfs_result: <function os._pickle_statvfs_result>,
 posix.stat_result: <function os._pickle_stat_result>}

Now, type of lambda functions is types.FunctionType. However, the builtin function for this type function: <function pickle.save_global> is not able to serialize lambda functions. Therefore, all third party libraries like dill, cloudpickle, etc override the inbuilt method to serialize lambda functions with some additional logic. Let's import dill and see what it does.

>> import dill
>> pickle.Pickler.dispatch

{_pyio.BufferedReader: <function dill.dill.save_file>,
 _pyio.TextIOWrapper: <function dill.dill.save_file>,
 _pyio.BufferedWriter: <function dill.dill.save_file>,
 _pyio.BufferedRandom: <function dill.dill.save_file>,
 functools.partial: <function dill.dill.save_functor>,
 operator.attrgetter: <function dill.dill.save_attrgetter>,
 operator.itemgetter: <function dill.dill.save_itemgetter>,
 cStringIO.StringI: <function dill.dill.save_stringi>,
 cStringIO.StringO: <function dill.dill.save_stringo>,
 bool: <function pickle.save_bool>,
 cell: <function dill.dill.save_cell>,
 instancemethod: <function dill.dill.save_instancemethod0>,
 instance: <function pickle.save_inst>,
 classobj: <function dill.dill.save_classobj>,
 code: <function dill.dill.save_code>,
 property: <function dill.dill.save_property>,
 method-wrapper: <function dill.dill.save_instancemethod>,
 dictproxy: <function dill.dill.save_dictproxy>,
 wrapper_descriptor: <function dill.dill.save_wrapper_descriptor>,
 getset_descriptor: <function dill.dill.save_wrapper_descriptor>,
 member_descriptor: <function dill.dill.save_wrapper_descriptor>,
 method_descriptor: <function dill.dill.save_wrapper_descriptor>,
 file: <function dill.dill.save_file>,
 float: <function pickle.save_float>,
 staticmethod: <function dill.dill.save_classmethod>,
 classmethod: <function dill.dill.save_classmethod>,
 function: <function dill.dill.save_function>,
 int: <function pickle.save_int>,
 list: <function pickle.save_list>,
 long: <function pickle.save_long>,
 dict: <function dill.dill.save_module_dict>,
 builtin_function_or_method: <function dill.dill.save_builtin_method>,
 module: <function dill.dill.save_module>,
 NotImplementedType: <function dill.dill.save_singleton>,
 NoneType: <function pickle.save_none>,
 xrange: <function dill.dill.save_singleton>,
 slice: <function dill.dill.save_slice>,
 ellipsis: <function dill.dill.save_singleton>,
 str: <function pickle.save_string>,
 tuple: <function pickle.save_tuple>,
 super: <function dill.dill.save_functor>,
 type: <function dill.dill.save_type>,
 weakcallableproxy: <function dill.dill.save_weakproxy>,
 weakproxy: <function dill.dill.save_weakproxy>,
 weakref: <function dill.dill.save_weakref>,
 unicode: <function pickle.save_unicode>,
 thread.lock: <function dill.dill.save_lock>}

Now, let's try to pickle lambda function.

>> pickle.loads(pickle.dumps(lambda x:x))
<function __main__.<lambda>>

It WORKS!!

In Python 2 we have two versions of pickle -

import pickle # pure Python version
pickle.__file__ # <install directory>/python-2.7/lib64/python2.7/pickle.py

import cPickle # C extension
cPickle.__file__ # <install directory>/python-2.7/lib64/python2.7/lib-dynload/cPickle.so

Now, let's try to pickle lambda with C implementation cPickle.

>> import cPickle
>> cPickle.loads(cPickle.dumps(lambda x:x))
TypeError: can't pickle function objects

What went wrong? Let's see the dispatch table of cPickle.

>> cPickle.Pickler.dispatch_table
AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch_table'

The implementation of pickle and cPickle is different. Importing dill makes only Python version of pickle work. The disadvantage of using pickle instead of cPickle is that it can be as much as 1000 times slower than cPickle.

Python 3.6

In Python 3, there is no module named cPickle. We have pickle instead which also doesn't support pickling of lambda functions by default. Let's see it's dispatch table:

>> import pickle
>> pickle.Pickler.dispatch_table
<member 'dispatch_table' of '_pickle.Pickler' objects>

Wait. I tried looking up dispatch_table of pickle not _pickle. _pickle is the alternative and faster C implementation of pickle. But we haven't imported it yet! This C implementation is imported automatically, if it is available, at the end of pure Python pickle module.

# Use the faster _pickle if possible
try:
    from _pickle import (
        PickleError,
        PicklingError,
        UnpicklingError,
        Pickler,
        Unpickler,
        dump,
        dumps,
        load,
        loads
    )
except ImportError:
    Pickler, Unpickler = _Pickler, _Unpickler
    dump, dumps, load, loads = _dump, _dumps, _load, _loads

We are still left with the question of pickling lambdas in Python 3. The answer is you CAN'T with the native pickle or _pickle. You will need to import dill or cloudpickle and use that instead of the native pickle module.

>> import dill
>> dill.loads(dill.dumps(lambda x:x))
<function __main__.<lambda>>

I hope this clears all the doubts.

Can Python pickle lambda functions?

Related

Recent Posts