Is there any option to limit mongodb memory usage?

If you are running MongoDB 3.2 or later version, you can limit the wiredTiger cache as mentioned above.

In /etc/mongod.conf add the wiredTiger part

...
# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
  wiredTiger:
    engineConfig:
        cacheSizeGB: 1
...

This will limit the cache size to 1GB, more info in Doc

This solved the issue for me, running ubuntu 16.04 and mongoDB 3.2

PS: After changing the config, restart the mongo daemon.

$ sudo service mongod restart

# check the status
$ sudo service mongod status

Starting in 3.2, MongoDB uses the WiredTiger as the default storage engine. Previous versions used the MMAPv1 as the default storage engine.

  • With WiredTiger, MongoDB utilizes both the WiredTiger internal cache and the filesystem cache.
  • In MongoDB 3.2, the WiredTiger internal cache, by default, will use the larger of either: 60% of RAM minus 1 GB, or 1 GB.
  • For systems with up to 10 GB of RAM, the new default setting is less than or equal to the 3.0 default setting (For MongoDB 3.0, the WiredTiger internal cache uses either 1 GB or half of the installed physical RAM, whichever is larger). For systems with more than 10 GB of RAM, the new default setting is greater than the 3.0 setting.

to limit the wiredTriggered Cache Add following line to .config file :

wiredTigerCacheSizeGB = 1