How does nginx verify certificates (proxy_ssl_verify)?

Consider nginx's parameters proxy_ssl_verify and proxy_ssl_trusted_certificate that are used to secure the communication to an upstream server.

# downstream server: nginx.conf

stream {
  server {
    # ...

    proxy_ssl on;
    proxy_ssl_verify on;

    proxy_ssl_trusted_certificate /etc/nginx/ssl/upstream.example.com/chain.pem;
    proxy_ssl_verify_depth 2;

  }
}

I was under the impression that, for proxy_ssl on, nginx would verify the certificate sent by the upstream server (upstream.example.com) against the proxy_ssl_trusted_certificate.

However, I've encountered a problem where nginx can't establish a secure connection to the upstream server and reports an upstream SSL certificate verify error: (2:unable to get issuer certificate) while SSL handshaking to upstream, while verifying the certificate with openssl does work.

# fullchain.pem: certificate sent by the upstream server
# chain.pem: CA certificate for `proxy_ssl_trusted_certificate`

# openssl verify -CAfile chain.pem fullchain.pem
fullchain.pem: OK

This made me wonder how nginx actually does the verification. Is there a way to reproduce this in isolation in order to debug why the error would occur?

Documentation

https://nginx.org/en/docs/http/ngx_http_proxy_module.html

proxy_ssl_verify on | off;

Enables or disables verification of the proxied HTTPS server certificate.

proxy_ssl_verify_depth number;

Sets the verification depth in the proxied HTTPS server certificates chain.

proxy_ssl_trusted_certificate file;

pecifies a file with trusted CA certificates in the PEM format used to verify the certificate of the proxied HTTPS server.

As the documentation says, proxy_ssl_trusted_certificate expects the CA certificate(s) to be used to verify the upstream server's TLS certificate. You seem to have provided that upstream server's TLS certificate chain. Provide the CA certificate(s) instead.