Way to limit how much RAM an arbitrary process can take up?
A similar question to this is noted here How Can I Limit the Amount of RAM a Mac OS Application Can Use?
They recommend using the ulimit command, but they were unable to successfully limit the amount to what they desired. I think the problem that they ran into was the fact that the option being set -u or -v was not accounting for the memory paging system, eg you can only allocate X amount of space for you process in real RAM, but we can page you out to virtual memory all day long.
Note that this may be Dangerous disabling Virtual Memory, it could result in your Mac just locking up, etc when it runs of of RAM.
Try running ulimit with arguments -l, -m, or -v however as noted in the ulimit man page it up to the OS if it will pay attention to these settings or not as noted.
ulimit: ulimit [-SHacdefilmnpqrstuvx] [limit]
Modify shell resource limits.
Provides control over the resources available to the shell and processes it creates, on systems that allow such control.
Can't really find any definite source that confirms if OS X ignores ulimit or not, but after the referenced ulimit fail
If you can get ulimit to work for your needs by itself then give it a go, if not a combination of ulimit and or disabling virtual memory may be able to let you set up that 3 GB limit on select processes. FYI here is a good link for more information onCompletely Disable Virtual Memory in Snow Leopard System 10.6
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist
If you are running Lion (or newer OSX) Server on your Mac, you might be able to run OSX inside a VM (VirtualBox, Parallels, or VMWare, et.al.) on that Mac. Limit the VM to 3GB of physical RAM, and any app or process inside the VM will also be so limited. The system hosting the VM should stay responsive even if the OS in the VM is backing-swap-disk-limited or crashes.
If you are debugging a potentially runaway process, you might be able to ssh into the VM to run the debugger or a debug nub from the command line, and log stuff up until things lock up.