Why is my server's memory usage so high with just 5 players online?

I see a number of possible problems that may help your server run with less ram.

PlayerViewDistance

On many servers, including my own, the ViewDistance parameter in your server properties file may be too high. Decreasing this value will cause less chunks to be loaded for the players, but if you set it too low, ie below 3, then you won't be able to see far enough to have a good time playing.

This is easily fixed by decreasing the "view-distance" parameter in the server.properties file.

Command line args

I see a small problem with the one you pasted in your question:

Java -Xms512M -X3584M -jar minecraft_server.jar

I could not find a java argument that is simply 'X', although I assume that you are trying to set the upper memory limit to 3.5GB. Here is what that should look like:

Java -Xms512M -Xmx3584M -jar minecraft_server.jar

Or an even simpler version:

Java -Xms512M -Xmx3.5G -jar minecraft_server.jar

If you're willing to experiment, lower the maximum limit by 512mb every restart, and measure how it performs. In some odd cases, you may see a decrease in actual used memory.

Use Java 7

From my past experience, running on an Ubuntu server, running a Craftbukkit server under Java7 instead of Java6 used a noticeably lesser amount of RAM.

Switch to Bukkit

Although this may be an unwanted option for you, it definitely works. With Craftbukit there is a multitude of Plugins available for download on http://bukkit.org/ that help to increase server performance and lower resource usage.