Hitting Maximum Recursion Depth Using Pickle / cPickle

Solution 1:

From the docs:

Trying to pickle a highly recursive data structure may exceed the maximum recursion depth, a RuntimeError will be raised in this case. You can carefully raise this limit with sys.setrecursionlimit().

Although your trie implementation may be simple, it uses recursion and can lead to issues when converting to a persistent data structure.

My recommendation would be continue raising the recursion limit to see if there is an upper bound for the data you are working with and the trie implementation you are using.

Other then that, you can try changing your tree implementation to be "less recursive", if possible, or write an additional implementation that has data persistence built-in (use pickles and shelves in your implementation). Hope that helps

Solution 2:

Pickle does need to recursively walk your trie. If Pickle is using just 5 levels of function calls to do the work your trie of depth 638 will need the level set to more than 3000.

Try a much bigger number, the recursion limit is really just there to protect users from having to wait too long if the recursion falls in an infinite hole.

Pickle handles cycles ok, so it doesn't matter even if your trie had a cycle in there

Solution 3:

Stack size must also be increased with resource.setrlimit to prevent segfault

If you use just sys.setrecursionlimit, you can still segfault if you reach the maximum stack size allowed by the Linux kernel.

This value can be increased with resource.setrlimit as mentioned at: Setting stacksize in a python script

import pickle
import resource
import sys

print resource.getrlimit(resource.RLIMIT_STACK)
print sys.getrecursionlimit()

max_rec = 0x100000

# May segfault without this line. 0x100 is a guess at the size of each stack frame.
resource.setrlimit(resource.RLIMIT_STACK, [0x100 * max_rec, resource.RLIM_INFINITY])
sys.setrecursionlimit(max_rec)

a = []
# 0x10 is to account for subfunctions called inside `pickle`.
for i in xrange(max_rec / 0x10):
    a = [a]
print pickle.dumps(a, -1)

See also: What is the maximum recursion depth in Python, and how to increase it?

The default maximum value for me is 8Mb.

Tested on Ubuntu 16.10, Python 2.7.12.