Limit jvm process memory on ubuntu

You can't control what you want to control, -Xmx only controls the Java Heap, it doesn't control consumption of native memory by the JVM, which is consumed completely differently based on implementation.

From the following article Thanks for the Memory ( Understanding How the JVM uses Native Memory on Windows and Linux )

Maintaining the heap and garbage collector use native memory you can't control.

More native memory is required to maintain the state of the memory-management system maintaining the Java heap. Data structures must be allocated to track free storage and record progress when collecting garbage. The exact size and nature of these data structures varies with implementation, but many are proportional to the size of the heap.

and the JIT compiler uses native memory just like javac would

Bytecode compilation uses native memory (in the same way that a static compiler such as gcc requires memory to run), but both the input (the bytecode) and the output (the executable code) from the JIT must also be stored in native memory. Java applications that contain many JIT-compiled methods use more native memory than smaller applications.

and then you have the classloader(s) which use native memory

Java applications are composed of classes that define object structure and method logic. They also use classes from the Java runtime class libraries (such as java.lang.String) and may use third-party libraries. These classes need to be stored in memory for as long as they are being used. How classes are stored varies by implementation.

I won't even start quoting the section on Threads, I think you get the idea that -Xmx doesn't control what you think it controls, it controls the JVM heap, not everything goes in the JVM heap, and the heap takes up way more native memory that what you specify for management and book keeping.

Like any other process on Linux, you can limit the memory usage of the JVM process (say to 4 GB):

$ ulimit -v 4194304
$ java ...   # execute your Java program

(Technically this is actually virtual memory, which is an upper bound for actual memory usage. But ulimit apparently doesn't work with actual RAM usage.)

This will cause memory allocations above the limit to fail, presumably resulting in an OutOfMemoryError and your application exiting.