How to limit the "resident set size" per user on Linux
Solution 1:
Limiting the total vmem size (-v) is probably what you want.
Solution 2:
ulimit -m never limited RSS per user.
In the past (Linux 2.4.x, x < 30), ulimit -m affected the RSS per process. And even then, it was advice to the memory manager, not a hard limit. (See the RLIMIT_RSS section of man setrlimit.)
Similarly, ulimit -v does not impose per user limits. It imposes per process limits.
If you want to impose per user limits on Linux, I recommend considering cgroups.