There is a similar question answered here: https://unix.stackexchange.com/questions/34334/how-to-create-a-user-with-limited-ram-usage

The limits imposed by ulimit and limits.conf is per process. I definitely wasn't clear on that point.

If you want to limit the total amount of memory a users uses (which is what you asked). You want to use cgroups.

In /etc/cgconfig.conf:

group memlimit {
    memory {
        memory.limit_in_bytes = 4294967296;
    }
}

This creates a cgroup that has a max memory limit of 4GiB.

In /etc/cgrules.conf:

luser   memory   memlimit/

This will cause all processes run by luser to be run inside the memlimit cgroups created in cgconfig.conf.