I have multiple instances of nginx, but not sure which one is "correct"

I inherited a production web server, and I see at least 5 copies of nginx on it:

/usr/local/nginx/sbin/nginx
/usr/local/nginx/sbin/nginx.old
/usr/local/nginx-new/sbin/nginx
/usr/local/nginx-nobrotli/sbin/nginx
/usr/sbin/nginx

The first 4 return "nginx version: nginx/1.10.0" when run with the "-v" option, whereas the last one returns "nginx version: nginx/1.0.15".

The server is serving live traffic so I have to be really careful making any changes.

From running ps aux | grep nginx I can see that the one currently being used is nginx-nobrotli:

nginx: master process /usr/local/nginx-nobrotli/sbin/nginx -c /etc/nginx/nginx.conf

The thing that has me confused is that when I just type nginx -t from the command line I get back errors. I think it's because I'm really invoking /usr/sbin/nginx -t (since nginx -v gives back "nginx version: nginx/1.0.15" I assume that's invoking the 5th one).

So here are the real questions:

how can I restart nginx and make sure it's using the same configuration that's currently running? (I'm freaked out that if I do a nginx -s reload it'll take down the site by loading the wrong version of nginx)
what's the best way to make the nginx -t command refer to the "correct" version of nginx? (assuming the one currently running is the correct one)
what's the difference between the configuration files I see in /etc/nginx/conf.d/ and the ones I see in the /usr/local/nginx*/conf/ folders?
if the /usr/local/nginx* folders all contain a binary under their respective sbin/ folders, does that mean they are the same? why would someone have 3 versions like this?

UPDATE 1:

The command nginx -t returns this:

nginx: [emerg] unknown directive "pagespeed" in /etc/nginx/conf.d/mainsite.conf-adminips-20170214:23
nginx: configuration file /etc/nginx/nginx.conf test failed

But the command /usr/local/nginx-nobrotli/sbin/nginx -t returns this:

nginx: the configuration file /usr/local/nginx-nobrotli/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx-nobrotli/conf/nginx.conf test is successful

This worries me a little because /usr/local/nginx-nobrotli/conf/nginx.conf is basically empty so I know it's not the right file to be checking. Should I run /usr/local/nginx-nobrotli/sbin/nginx -t -c /etc/nginx/nginx.conf to have the "-t" option check that file? I know that "/etc/nginx/nginx.conf" is the right file, I'm not sure how to #1: check that it's valid and #2: safely do a nginx reload that is sure to be using the new file.

Thank you.

UPDATE 2:

Here are the contents of /etc/init.d/nginx:

#!/bin/sh
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig:   - 85 15
# description:  Nginx is an HTTP(S) server, HTTP(S) reverse \
#               proxy and IMAP/POP3 proxy server
# processname: nginx
# config:      /etc/nginx/nginx.conf
# config:      /etc/sysconfig/nginx
# pidfile:     /var/run/nginx.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0

nginx="/usr/sbin/nginx"
prog=$(basename $nginx)

sysconfig="/etc/sysconfig/$prog"
lockfile="/var/lock/subsys/nginx"
pidfile="/var/run/${prog}.pid"

NGINX_CONF_FILE="/etc/nginx/nginx.conf"

[ -f $sysconfig ] && . $sysconfig


start() {
    [ -x $nginx ] || exit 5
    [ -f $NGINX_CONF_FILE ] || exit 6
    echo -n $"Starting $prog: "
    daemon $nginx -c $NGINX_CONF_FILE
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}

stop() {
    echo -n $"Stopping $prog: "
    killproc -p $pidfile $prog
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

restart() {
    configtest_q || return 6
    stop
    start
}

reload() {
    configtest_q || return 6
    echo -n $"Reloading $prog: "
    killproc -p $pidfile $prog -HUP
    echo
}

configtest() {
    $nginx -t -c $NGINX_CONF_FILE
}

configtest_q() {
    $nginx -t -q -c $NGINX_CONF_FILE
}

rh_status() {
    status $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

# Upgrade the binary with no downtime.
upgrade() {
    local oldbin_pidfile="${pidfile}.oldbin"

    configtest_q || return 6
    echo -n $"Upgrading $prog: "
    killproc -p $pidfile $prog -USR2
    retval=$?
    sleep 1
    if [[ -f ${oldbin_pidfile} && -f ${pidfile} ]];  then
        killproc -p $oldbin_pidfile $prog -QUIT
        success $"$prog online upgrade"
        echo 
        return 0
    else
        failure $"$prog online upgrade"
        echo
        return 1
    fi
}

# Tell nginx to reopen logs
reopen_logs() {
    configtest_q || return 6
    echo -n $"Reopening $prog logs: "
    killproc -p $pidfile $prog -USR1
    retval=$?
    echo
    return $retval
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart|configtest|reopen_logs)
        $1
        ;;
    force-reload|upgrade) 
        rh_status_q || exit 7
        upgrade
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    status|status_q)
        rh_$1
        ;;
    condrestart|try-restart)
        rh_status_q || exit 7
        restart
        ;;
    *)
        echo $"Usage: $0 {start|stop|reload|configtest|status|force-reload|upgrade|restart|reopen_logs}"
        exit 2
esac

This tells me that if I were to reboot my server, my webserver won't start, since the executable at /usr/sbin/nginx (mentioned in the startup script) fails to parse the config when I run /usr/sbin/nginx -t. Is that correct?

A cowboy was here

A Cowboy was Here

nginx: [emerg] unknown directive "pagespeed"

This is why you have a copy of nginx in /usr/local, the pagespeed module has to be compiled into nginx. Good intentions (ahem^*) explains multiple copies.

Nginx 1.0.15 is old (from 4/2012) so it's unlikely the system you have is using systemd so look for init scripts in /etc/init.d.

What I would do in this situation is create a VM of the same base OS and copy the following directory trees to it.

/etc/nginx
/usr/local/nginx-nobrotli/
Your production tree
relevant init scripts from /etc/init.d

Don't install nginx from your distro.

Now you have a test environment, you can figure stuff out.

Add /usr/local/nginx-nobrotli/sbin/nginx to your PATH

export /usr/local/nginx-nobrotli/sbin/nginx:$PATH

Now when you run nginx it will be running the locally compiled version, it should work as you expect. It may still throw curved balls depending on how it was compiled, for example you may need to use -c /etc/nginx/nginx.conf to ensure that it is picking up the correct config file. You could fix that with an alias.

How can I restart nginx ...

hopefully /etc/init.d/nginx restart (or similar) you'll have to figure that out based on what you find in your environment.

What's the best way to make the nginx -t ...

Remove all irrelevant versions (based upon testing) and adjust the PATH appropriately. See above regarding the config options.

what's the difference between ...

They are likely irrelevant given the output of your ps command. However in general they would be where the locally compiled nginx would expect to find it's config. We know that they are likely not in use from the ps command.

if the /usr/local/nginx* folders all contain a binary under their respective sbin/ folders, does that mean they are the same?

Not guaranteeably they may have differing compilation options.

why would someone have 3 versions like this?

Good intentions, testing, version control some other reason - you choose.

When you have figured it out document it. Go back to your prod system and remove the cruft. Negotiate a maintenance window and test what you learned and implemented.

^*A Cowboy. I'm sure many people will recognise this, the better of us grow out of it.

I have multiple instances of nginx, but not sure which one is "correct"

Related

Recent Posts