MongoDB getting OOM killed

we are running a mongodb replicaset on three machines. All three machines have around 16GB but only 255MB Swap. Swappiness is left on it's default value 60. The machines are running CentOS 6.4. The databases are much larger than the 16GB, but that's ok for us. The really working set is much smaller.

The problem we are facing is that the primary consumes eats up all available memory and than getting OOM-Killed. I know that this is the way how mongodb manages memory.

After the server is getting OOM killed someone has to manually restart it.

Is there any way to prevent mongodb from getting OOM killed? Adjust the swappiness? Increase swap space? I think that those settings will only increase the grace period before mongod gets killed.

OOM killer is not a way anyone manages memory; it is Linux kernels way to handle fatal failure in last hope to avoid system lockup!

What you should do is:

• make sure you have enough swap. If you are sure, still add more.

• implement resource limits! At LEAST for applications you expect that will use memory (and even more so if you don't expect them to - those ones usually end up being problematic). See ulimit -v (or limit addressspace) commands in your shell and put it before application startup in its init script. You should also limit other stuff (like number of processes -u, etc)... That way, application will get ENOMEM error when there is not enough memory, instead of kernel giving them non-existent memory and afterwards going berserk killing everything around!

• tell the kernel not to overcommit memory. You could do:

  echo "0" > /proc/sys/vm/overcommit_memory

  or even better (depending on your amount of swap space)

  echo "2" > /proc/sys/vm/overcommit_memory; echo "80" > /proc/sys/vm/overcommit_ratio

  See Turning off overcommit for more info on that.

  That would instruct kernel to be more carefull when giving applications memory it doesn't really have (similarity to worlds global economic crisis is striking)

• as a last dich resort, if everything on your system except MangoDB is expendable (but please fix two points above first!) you can make lower the chances of it being killed (or even making sure it won't be killed - even if alternative is hangup machine with nothing working) by tuning /proc/$pid/oom_score_adj and/or /proc/$pid/oom_score.

  echo "-1000" > /proc/`pidof mangod`/oom_score_adj

  See Taming the OOM killer for more info on that subject.