How should I start MongoDB on a NUMA machine?

When I first installed MongoDB 2.2.3 with apt-get, following the instructions from 10gen installation guide, it auto starts when the server starts.

However, after following the instructions in /var/log/mongodb/mongodb.log:

Tue Apr 30 11:35:28.643 [initandlisten] ** WARNING: You are running on a NUMA machine.
Tue Apr 30 11:35:28.643 [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
Tue Apr 30 11:35:28.643 [initandlisten] **              numactl --interleave=all mongod [other options]
Tue Apr 30 11:35:28.643 [initandlisten]
Tue Apr 30 11:35:28.643 [initandlisten] ** WARNING: /proc/sys/vm/zone_reclaim_mode is 1
Tue Apr 30 11:35:28.643 [initandlisten] **          We suggest setting it to 0
Tue Apr 30 11:35:28.643 [initandlisten] **          http://www.kernel.org/doc/Documentation/sysctl/vm.txt
Tue Apr 30 11:35:28.643 [initandlisten]

it doesn't auto start any more.

So now I have to enter:

echo 0 > /proc/sys/vm/zone_reclaim_mode && numactl --interleave=all /usr/bin/mongod --config /etc/mongodb.conf &

manually every time in a terminal to start it.

However, I suspect starting it this way also causes service mongodb stop/restart to fail. It will throw a stop: Unknown instance: error.

Is there a proper way to resolve this?

Solution 1:

Upstart cannot stop or restart your service if you started it manually. You need to update the upstart script for mongodb so that it starts with the options you require, and then start it with service mongodb start.

First of all, since the zone_reclaim_mode setting is system-wide rather than specific to mongodb, you could add the line:

vm.zone_reclaim_mode = 0

to /etc/sysctl.conf, which configures system variables on startup.

Next, you need to update the script that upstart uses to start mongodb, /etc/init/mongodb.conf, so that it wraps the normal command with the numactl command. So the line:

if [ "x$ENABLE_MONGODB" = "xyes" ]; then exec start-stop-daemon --start --quiet --chuid mongodb --exec  /usr/bin/mongod -- --config /etc/mongodb.conf; fi

becomes:

if [ "x$ENABLE_MONGODB" = "xyes" ]; then exec start-stop-daemon --start --quiet --chuid mongodb --exec  /usr/bin/numactl -- --interleave=all /usr/bin/mongod --config /etc/mongodb.conf; fi

You also mention that upstart won't auto-start your service on boot; if that still happens after making the above changes, please attached a full extract from /var/log/mongodb/mongodb.log of a failed start.

Solution 2:

for ubuntu 16.04 mongod.service file should look like

[Unit]
Description=High-performance, schema-free document-oriented database
After=time-sync.target network.target

[Service]
Type=forking
User=mongod
Group=mongod
LimitNOFILE=65000
PermissionsStartOnly=true
EnvironmentFile=/etc/default/mongod
ExecStartPre=/usr/bin/percona-server-mongodb-helper.sh
ExecStart=/usr/bin/env bash -c "numactl --interleave=all /usr/bin/mongod $OPTIONS > ${STDOUT} 2> ${STDERR}"
#ExecStart=/usr/bin/env bash -c "/usr/bin/mongod $OPTIONS > ${STDOUT} 2> ${STDERR}"
PIDFile=/var/run/mongod.pid

[Install]
WantedBy=multi-user.target

and you also can specify nofile limit with LimitNOFILE param

Solution 3:

The Ubuntu MongoDB upstart script now checks for the presence of the numactl binary and adjusts accordingly.

The NUMA warning can be fixed as follows with Ubuntu 14.04 LTS and MongoDB 2.6:

sudo apt-get -y install numactl
sudo restart mongod

Solution 4:

i know the question is about upstart, but just in case someone (like me) needs a init.d solution:

replace the start_server() function in /etc/init.d/mongodb for the code below

start_server() {
    test -e "$RUNDIR" || install -m 755 -o mongodb -g mongodb -d "$RUNDIR"

    NUMACTL=$(which numactl)

    if [ ! "$NUMACTL" ]; then
        # start original
        start-stop-daemon --background --start --quiet --pidfile $PIDFILE --make-pidfile --chuid $DAEMONUSER --exec $DAEMON -- $DAEMON_OPTS
        errcode=$?
        return $errcode
    else
        # Start the process using the wrapper
        $NUMACTL --interleave=all -- start-stop-daemon --background --start --quiet --pidfile $PIDFILE --make-pidfile --chuid $DAEMONUSER --exec $DAEMON -- $DAEMON_OPTS
        errcode=$?
        return $errcode
    fi
}