How to set the maximum memory usage for JVM?

use the arguments -Xms<memory> -Xmx<memory>. Use M or G after the numbers for indicating Megs and Gigs of bytes respectively. -Xms indicates the minimum and -Xmx the maximum.

You shouldn't have to worry about the stack leaking memory (it is highly uncommon). The only time you can have the stack get out of control is with infinite (or really deep) recursion.

This is just the heap. Sorry, didn't read your question fully at first.

You need to run the JVM with the following command line argument.

-Xmx<ammount of memory>

Example:

-Xmx1024m

That will allow a max of 1GB of memory for the JVM.

If you want to limit memory for jvm (not the heap size ) ulimit -v

To get an idea of the difference between jvm and heap memory , take a look at this excellent article http://blogs.vmware.com/apps/2011/06/taking-a-closer-look-at-sizing-the-java-process.html

The answer above is kind of correct, you can't gracefully control how much native memory a java process allocates. It depends on what your application is doing.

That said, depending on platform, you may be able to do use some mechanism, ulimit for example, to limit the size of a java or any other process.

Just don't expect it to fail gracefully if it hits that limit. Native memory allocation failures are much harder to handle than allocation failures on the java heap. There's a fairly good chance the application will crash but depending on how critical it is to the system to keep the process size down that might still suit you.

The NativeHeap can be increasded by -XX:MaxDirectMemorySize=256M (default is 128)

I've never used it. Maybe you'll find it useful.