Minecraft Server keeps crashing out of memory, but stil empty swap left
You can allocate manually how much RAM java can use for minecraft. From Minecraft Wiki
Using the Xms and Xmx parameters, the initial and maximum memory size for Java can be specified. By default, your server runs with about 100 MB of RAM, which is very little. Most people will change their server to run with more, for example:
java -Xms512M -Xmx1G ... java -Xms1024M -Xmx1024M ... java -Xms1G -Xmx2G ...