How to set a memory limit for a specific process?

I tried to execute mpiexec -16 ... on a 384 GB RAM server, but it induced OOM Killer and was aborted.

How can I set a memory limit for mpiexec execution?

I know ulimit, but it might affect other processes.

Thank you.

I think this can be done using cgroups:

Create a cgroup named mpigroup (or whatever name you choose) with a memory limit (of 50GB, for example):

cgcreate -g memory,cpu:mpigroup
cgset -r memory.limit_in_bytes=$((50*1024*1024*1024)) mpigroup

Then, if mpiexec is already running, bring it into this cgroup:

cgclassify -g memory,cpu:mpigroup $(pidof mpiexec)

Or execute mpiexec within this cgroup:

cgexec -g memory,cpu:mpigroup mpiexec -16 ...