How do I limit the memory usage of a specific process in linux?

You'll want to use ulimit

ulimit can be used to limit memory utilization (among other things)

Here is an example of setting memory usage so low that /bin/ls (which is larger than /bin/cat) no longer works, but /bin/cat still works.

$ ls -lh /bin/ls /bin/cat
-rwxr-xr-x 1 root root 25K May 24 2008 /bin/cat
-rwxr-xr-x 1 root root 88K May 24 2008 /bin/ls
$ date > test.txt
$ ulimit -d 10000 -m 10000 -v 10000
$ /bin/ls date.txt
/bin/ls: error while loading shared libraries: libc.so.6: failed to map segment from shared object: Cannot allocate memory
$ /bin/cat date.txt
Thu Mar 26 11:51:16 PDT 2009
$

Note: If I set the limits to 1000 kilobytes, neither program works, because they load libraries, which increase their size. above 1000 KB.

-d data segment size

-m max memory size

-v virtual memory size

Run ulimit -a to see all the resource caps ulimits can set.