Why does the Java VM process eat up more RAM then specified in -Xmx parameter?
-Xmx is the maximum amount of heap space being used by the vm. It does not include memory for the classes loaded, the vm itself or stacks of threads or the memory used for the JIT for instance. The line from top is also misleading since the first number included shared memory that's used by other programs as well. The second nummer 3.9g is the resident size and closer to reality.
My recommendation: Check if you really need that much heap (number and size of objects) and reduce the -Xmx number if you can.