MongoDB best practice against constant power failure

Background:

I have some AIO application machines, (running ubuntu with rails + mongodb + chromium) in one machine on a mobile coach for displaying information, occasionally user might change chuck in some data into mongodb.

Now, the power supply of the machine is constant, coach driver may power off the machine at any time. machine has 2 partition, first is aufs for /, the other is ext4 for mongodb /data directory. BUT during the instant of power cut, there is no data input into the db.

Problem:

Every time the machine is restarted due to power cut (every several hours), mongodb will leave a mongod.lock file in its directory. in /etc/rd.local I have tried to delete the lock file every boot up, but it sometimes still refuse to start. which causing my app fail the start.

According to official docs: http://www.mongodb.org/display/DOCS/Durability+and+Repair , I still have some occasion unable to start.

What's the best practice to run mongodb in a regular power failure situation like above? Without throwing additional hardware.

Solution 1:

All the mongod.lock file is telling you is that the DB had an unclean shutdown, i.e. was not stopped by an admin, etc.

When running a --repair operation by itself, mongod will attempt to read the existing files, write new files and then swap them. Once complete, it should remove the mongod.lock file, and enable you to startup the database.

If used in conjunction with --repairpath argument, then the repaired filed will be placed at the specified repair path, and the lock file may not be removed, as the original data files have not been repaired, rather new files have been written with the repaired data and the specified path.

A possible flow of working with --repairpath:

1. Service startup, log message informs you of lock problem, exits.

   mongod --dbpath=/data/db
   

2. You run a repair command similar to this:

   mongod --dbpath=/data/db --repair --repairpath=/data/db2
   

   and wait for it to complete.

3. Once complete, start up mongod from the repaired files path:

   mongod --dbpath=/data/db2
   

4. Once confirmed working, you can remove /data/db directory if you desire.

All of this can probably be eliminated by "replacing" the files in /data/db/ by not using the --repairpath option.

In regards to recovery, take a look at Journaling - this creates an operation journal that flushes to persistent disk every 100ms, and when mongod starts up and detects unapplied journal files and applies them, removes the lock file, and starts up the server.

Solution 2:

First of all, please READ the blessed documentation you linked to:

mongod.lock
Do not remove the mongod.lock file. If mongod is unable to start, use one of the methods above to correct the situation.

Removing the lock file will allow the database to start when its data may be corrupt. In general, you should never force the database to start with possibly corrupt data. In an emergency situation, you may want to remove the lock file to pull whatever data you can off the server. If you have ever manually removed the lock file and started the server back up, you should not consider that server "healthy."

Follow the procedures outlined in the blessed documentation you linked to to recover after power failures.

MongoDB isn't designed to be run in an environment where it will be unceremoniously kicked in the head -- If you need that kind of durability you need a database specifically designed for two-phase commit and flushed writes to disk.