Two Way SSL Error - 400 The SSL certificate error just for client certificate

Finally, I have pinned down the root cause of the problem. There were two problems with my setup.

a) For two-way SSL, the certificate signed by the Intermediate CA must have clientAuth in extendedKeyUsage (Thanks to @dave_thompson_085) which can be verified by the below command

$ openssl x509 -in /path/to/client/cert -noout -purpose | grep 'SSL client :'
SSL client : Yes

b) Another, thing which was missing was ssl_verify_depth parameter in the nginx config file which must be 2 or more. It does not make much sense to make the number bigger than 2 in my case, but it works with any number other than 1 (which is default value). Interestingly, this is not required in nginx v1.12.X (my colleague with the exact same setup didn't have to specify this). However, it didn't work for me (nginx v1.13.5) until I used this parameter.

I can have a sound sleep after 3 days of headbanging.

TIP: Don't depend on curl much to troubleshoot two-way SSL issues, try openssl s_client instead. curl can give misleading results sometimes, see this. I too fumbled around for a while in my Ubuntu 16.04 docker container.

openssl s_client -cert $file can provide only 'the' client cert (singular, one), not a chain; it can optionally provide the privatekey as well, if you don't specify -key, but still not any cert(s) other than the client cert. If you put any additional cert(s) in that file they are totally and completely ignored.

But libssl uses certs from the truststore to fill out the client's cert chain in addition to its nominal purpose of validating the server's cert chain. Read the subtly definitive words of the man page:

-CApath directory

The directory to use for server certificate verification. This directory must be in "hash format", see verify for more information. These are also used when building the client certificate chain.

-CAfile file

A file containing trusted certificates to use during server authentication and to use when attempting to build the client certificate chain.

Since you need to specify your private root to verify the server chain, if you instead give -CAfile a file containing both your root and your intermediate then both directions will work.

Unlike some other implementations where putting an intermediate in the truststore causes validation to stop at that point, OpenSSL will use truststore certs to complete a received chain that is incomplete, but by default still validates up to a root from the truststore; to stop earlier you must specify -partial_chain (new in 1.1.0).

However, I'm surprised configuring the server with ssl_client_certificate containing root and intermediate didn't work. nginx also uses OpenSSL and as I said libssl will use intermediate(s) from truststore to complete (though not validate) a received chain. So that approach, although officially not standard-conforming (the sender is responsible for sending a full chain modulo root) should actually work. I'll try to test later when I have a chance.