How to limit memory of a OS X program? ulimit -v neither -m are working

You can't. Apple can (using the ledger() system call, which is private), but you can't. I'm not entirely sure whether launchd's options work or not - certainly if it was still using the code visible in the last open source version (from 10.9.5), it wouldn't, because it calls setrlimit(), but it's been substantially rewritten since then, though I can't see it calling ledger(), which I'd expect it to if this was supposed to work.

Why? Because the RLIMIT_DATA and RLIMIT_AS options to setrlimit() don't actually do anything in current versions of XNU (the macOS kernel).

After struggling with this myself (with limited success), I have determined there seems to be two ways to do it...

You can setup a launchd item for your executable.. The important part of the plist is a section, such as..

<key>SoftResourceLimits</key>
<dict>
    <key>Stack</key>
    <integer>10000000000</integer>
</dict>

There are various keys available... which can be found on Apple's MAN page.

Another way to do it, I think, is by setting a value in either /etc/launchd.conf (system) or /etc/launchd-usr.conf (peruser). For example, your launchd.conf could contain...

umask 002
limit stack 67104768 67104768
limit maxproc 3400 4500
limit maxfiles 256 unlimited
setenv PATH /opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin

The documentation for all of launchd's functionality is shotty, if you ask me.. It's as if Apple might not care / want people outside their walls to actually understand how it all works. There is so much power to be had by mastering launchd and it's intricacies... but there are few concrete/official resources available as to how to properly implement them.

setrlimit should do the job. I believe that's the BSD equivalent of ulimit...