Get Python to raise MemoryError instead of eating all my disk space

If I run a Python program with a memory leak, I would normally expect the program to eventually die with MemoryError. But instead, what happens is that all the virtual memory is used until my disk runs out of space. I am running Mac OS X 10.8 on a retina MacBook Pro. My computer generally has between 10GB to 20GB free. Mac OS X is smart enough to not die completely when the disk runs out of space (rather, it gives me a dialog letting me force quit my GUI programs).

Is there a way to make Python just die when it runs out of real memory, or some reasonable amount of virtual memory? This is what happens on Linux, as far as I can tell. I guess Mac OS X is more generous than Linux with virtual memory (the fact that I have an SSD might be part of this; I don't know just how smart OS X is with this stuff). Maybe there's a way to tell the Mac OS X kernel to never use so much virtual memory that leaves less than, say, 5 GB free on the hard drive?

Solution 1:

Python Level

According to this post, resource.setrlimit() maybe what you need.

Example

#!/usr/bin/python

import resource
import sys
import signal
import time

import os

soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
print 'Soft limit starts as  :', soft

# Use env MY_PY_SET_LIMIT to control limit value
# If MY_PY_SET_LIMIT is not set, RLIMIT_STACK will not change
MY_PY_SET_LIMIT = os.getenv('MY_PY_SET_LIMIT')

if MY_PY_SET_LIMIT != None :
  resource.setrlimit(resource.RLIMIT_STACK, (int(MY_PY_SET_LIMIT), int(MY_PY_SET_LIMIT)))

soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
print 'Soft limit changed to :', soft

TMP = ""

for i in range(10240):
  TMP += "0123456789"
  print len(TMP)

System Level

For Linux, it is actually answer multiple times before on various "board" of stackexchange and other sites too. The best answer I found is here which contain an example.

The answer is use ulimit -v < kByte >. For example, limiting the vm to 10M:

ulimit -v 10240

However, on OS X there are indication (here & here) that ulimit maybe ignore. Those links are very old. I am not sure if situation change in more recent OS X releases.

There is this post for OS X to use launchd conf. It suggest using a Stack section in a plist config

<key>SoftResourceLimits</key>
<dict>
    <key>Stack</key>
    <integer>10000000000</integer>
</dict>

Or with /etc/launchd.conf

launchd.conf

umask 002
limit stack 67104768 67104768
limit maxproc 3400 4500
limit maxfiles 256 unlimited
setenv PATH /opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin

PS: according to Mountain Lion man launchd.conf(5) per user launchd.conf is not support

 $HOME/.launchd.conf  Your launchd configuration file (currently unsupported).