Why getting error mongod dead but subsys locked and Insufficient free space for journal files on Linux?

I have installed mongo-10gen mongo-10gen-server on Linux CentOS server.

I followed the steps from Link.

I have configured /etc/mongod.conf as -

logpath=/var/log/mongo/mongod.log
port=27017
dbpath=/var/lib/mongo

I have set port 27017 for mongo in iptables. To start mongo I used commands -

service mongod start and
mongo

It get started well, but after few days I am getting the error -

Tue Jan 29 08:41:54 [initandlisten] ERROR: Insufficient free space for journal files
Tue Jan 29 08:41:54 [initandlisten] Please make at least 3379MB available in /var/lib/mongo/journal or use --smallfiles
Tue Jan 29 08:41:54 [initandlisten]
Tue Jan 29 08:41:54 [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
Tue Jan 29 08:41:54 dbexit:
Tue Jan 29 08:41:54 [initandlisten] shutdown: going to close listening sockets...
Tue Jan 29 08:41:54 [initandlisten] shutdown: going to flush diaglog...
Tue Jan 29 08:41:54 [initandlisten] shutdown: going to close sockets...
Tue Jan 29 08:41:54 [initandlisten] shutdown: waiting for fs preallocator...
Tue Jan 29 08:41:54 [initandlisten] shutdown: lock for final commit...
Tue Jan 29 08:41:54 [initandlisten] shutdown: final commit...
Tue Jan 29 08:41:54 [initandlisten] shutdown: closing all files...
Tue Jan 29 08:41:54 [initandlisten] closeAllFiles() finished
Tue Jan 29 08:41:54 [initandlisten] journalCleanup...
Tue Jan 29 08:41:54 [initandlisten] removeJournalFiles
Tue Jan 29 08:41:54 [initandlisten] shutdown: removing fs lock...
Tue Jan 29 08:41:54 dbexit: really exiting now

When I execute the command -

service mongod status

It gives Error -

mongod dead but subsys locked

Please help me to solve the problem of mongod dead but subsys locked and Insufficient free space for journals, terminating

Solution 1:

You can add following to the config file provided when running mongod --config mongod.conf

For MongoDB 3.x (latest version)

storage:
   mmapv1:
      smallFiles: true

For version 2.6+

storage:
   smallFiles: true

For version 2.4 and less

smallfiles = true

Then just execute mongod to accept your config file (here it assumes that location of the config is /etc/mongodb.conf):

mongod -f /etc/mongodb.conf

Documentation for smallfiles parameter:

Set to true to modify MongoDB to use a smaller default data file size. 
Specifically, smallfiles reduces the initial size for data files and
limits them to 512 megabytes. The smallfiles setting also reduces the
size of each journal files from 1 gigabyte to 128 megabytes.

Solution 2:

Start mongod instance using following command

mongod --dbpath /data/db --smallfiles

Solution 3:

I was following the official guide at http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/

After using

$sudo service mongod start

I went to verify if MongoDB had started successfully by reviewing this log:

/var/log/mongodb/mongod.log

This is the problem that I found:

2014-11-11T12:54:05.808-0500 [initandlisten] ERROR: Insufficient free space for journal files
2014-11-11T12:54:05.808-0500 [initandlisten] Please make at least 3379MB available in /var/lib/mongodb/journal or use --smallfiles
2014-11-11T12:54:05.808-0500 [initandlisten]
2014-11-11T12:54:05.808-0500 [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
2014-11-11T12:54:05.808-0500 [initandlisten] dbexit:
2014-11-11T12:54:05.808-0500 [initandlisten] shutdown: going to close listening sockets...
2014-11-11T12:54:05.808-0500 [initandlisten] shutdown: going to flush diaglog...
2014-11-11T12:54:05.808-0500 [initandlisten] shutdown: going to close sockets...
2014-11-11T12:54:05.808-0500 [initandlisten] shutdown: waiting for fs preallocator...
2014-11-11T12:54:05.808-0500 [initandlisten] shutdown: lock for final commit...
2014-11-11T12:54:05.808-0500 [initandlisten] shutdown: final commit...
2014-11-11T12:54:05.808-0500 [initandlisten] shutdown: closing all files...
2014-11-11T12:54:05.808-0500 [initandlisten] closeAllFiles() finished
2014-11-11T12:54:05.808-0500 [initandlisten] journalCleanup...
2014-11-11T12:54:05.808-0500 [initandlisten] removeJournalFiles
2014-11-11T12:54:05.814-0500 [initandlisten] shutdown: removing fs lock...
2014-11-11T12:54:05.814-0500 [initandlisten] dbexit: really exiting now

Solution:

At the end of the file /etc/mongod.conf I added this line:

smallfiles = true

After that, I restarted the mongod service:

$sudo service mongod restart

Then when I went to review the log, I realized that everything was perfect, and the problem was fixed:

2014-11-11T22:32:20.544-0500 ***** SERVER RESTARTED *****
2014-11-11T22:32:20.552-0500 [initandlisten] MongoDB starting : pid=5200 port=27017 dbpath=/var/lib/mongodb 64-bit host=jaimemontoya-VirtualBox
2014-11-11T22:32:20.552-0500 [initandlisten] db version v2.6.5
2014-11-11T22:32:20.552-0500 [initandlisten] git version: e99d4fcb4279c0279796f237aa92fe3b64560bf6
2014-11-11T22:32:20.552-0500 [initandlisten] build info: Linux build8.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2014-11-11T22:32:20.552-0500 [initandlisten] allocator: tcmalloc
2014-11-11T22:32:20.552-0500 [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, storage: { dbPath: "/var/lib/mongodb", smallFiles: true }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2014-11-11T22:32:20.564-0500 [initandlisten] journal dir=/var/lib/mongodb/journal
2014-11-11T22:32:20.564-0500 [initandlisten] recover : no journal files present, no recovery needed
2014-11-11T22:32:20.738-0500 [initandlisten] waiting for connections on port 27017
2014-11-11T22:33:20.748-0500 [clientcursormon] mem (MB) res:36 virt:245
2014-11-11T22:33:20.748-0500 [clientcursormon]  mapped (incl journal view):64
2014-11-11T22:33:20.748-0500 [clientcursormon]  connections:0