Severe system lag spike when running Minecraft server with more than 1 user?

I am trying to figure out what the problem could be with my game server. Currently it only run a Minecraft server but may run more later.

The problem I am experencing is severe system lag spikes when there is more than 1 user connected to the at once. The lag spike do not just affect the minecraft instance they affect the WHOLE system, SSH becomes unresponsive and one time the machine went completely offline and I had to drop in with the management console to regain control.

The server is running the Feed the Beast Direwolf20 modpack server.

The Specs!

• Virtual Machine running on VMware vSphere 5.1
• 1 vCPU with 2 Cores
• Allocated 4GB of vRAM (out of 16GB available to the system)
• 20 GB VMDK (about 20% used)
• 1 Network Adapter (VMXNET3 I think)
• Running Ubuntu 12.10 (all latest updates)
• VMware Tools are current
• Java Runtime 1.7.0 u13 (current as of this post)

When I run top on the server it displays some really strange stats for the Java process which when minecraft is running sits at the top of the list which makes sense.

For example the server says Java is using 120% of the CPU and close to all of the ram assigned to Java at the start of Minecraft. Almost as soon as one user disconnects from the server the CPU drops to around 9.8% usage for the Java process. It just seems very odd.

The command I using to run the server is java -Xms512M -Xmx1024M -jar FTBServer.jar

There is only 1 other VM running on this server right now and its just BIND server for local network traffic.

Is there anything I could be missing? Something is very wrong but I am sure if its Java, the Ubuntu VM or VMware vSphere.

If anyone could be give me a hand that would be awesome.

I had originally asked on SeverFault and was recommended to ask here instead.

"Virtual Machine running on VMware vSphere 5.1"

There's your problem. For a user to connect the system has to process the VM and the system itself. It has to redirect all packets constantly to the VM. If you run it normally it should work.