How can I debug nginx further than the error log?

I'm assuming you've already jacked your Nginx error logging level up to debug. If not, start there.

Your best bet is probably going to be using strace to view the system calls being made by Nginx. In particular, you'll want to pay attention to connect() calls, and keep an eye on the return codes of these (man 2 connect can be your friend here).

Once you have that information, you can better make an educated guess about whether the issue is confined to your frontend proxy, or has something to do with the interactions between the proxy and backend application server.

It doesn't get much more pedantic than this unless you want to put in dtrace probes:

1. Set debug log level: /etc/nginx/nginx.conf:

   ...
   http {
           ...
           error_log /var/log/nginx/error.log debug; # todo testing remove me not for production use
           ...
   }
   

2. Setup tcpdump in another window:

   tcpdump not port 22 -vvv -s0 -q -XXX
   

3. Monitor log files in yet another window:

   tail -f /var/log/nginx/*
   

4. Startup nginx interactively with strace:

   # top of /etc/nginx/nginx.conf:
   
   daemon off; # todo testing remove me not for production use
   

   And then

    $ strace nginx 
   

Further debugging can be had with an nginx compiled with --with-debug. Check it by running:

    nginx -V 2>&1 | grep -- '--with-debug' # no output if not debug

Another good module not compiled by default is: HttpStubStatusModule. In all likelihood, any decent setup will require a custom-compiled nginx (highly-recommend packaging using distro's packaging tools).

Most of these are unsuitable for production use, look at compiling nginx with gperf if you need more stats.

Looks like you are debugging a high traffic site.

Use debug with debug_connection directive so nginx error log will show debug logs from your IP only.

Once you start seeing some useful error logs rather than activating debug option for entire nginx config, add a separate error_log /path/to/some/file/ debug; directive in location {..} block responsible for reverse_proxy connection.

This way you will be able to isolate debug error log from your IP only.

Try to relate it with request you are making (from your browser).

For example, please check: https://easyengine.io/tutorials/nginx/debugging/

A level ahead, you can use Nginx's HttpEchoModule

I've never found Nginx to be a bottleneck, in most cases its more than capable than the back ends. But if you tested without Nginx and found no error, then its going to be either (or both):

1. Nginx configuration issue
   1. Upstream time out value wrong
   2. Wrong probe URL on upstream
   3. Too few workers
   4. Etc.
2. Operating system TCP/IP bottleneck
   1. It could be that the proxying itself is causing a duplication of open ports and states. Be it file descriptors, ports, TCP connections

Without seeing your Nginx configs, no-one can comment on the former. And without suitable outputs from the OS, no-one can comment on the latter.