How does an SSL certificate chain bundle work?

I've created a chain hierarchy like this.

root-ca ==> signing-ca ==> subordinate-ca ==> server

It is mentioned to create chain bundle, the lowest should go first.

$ cat server.crt subordinate-ca.crt signing-ca.crt > server.pem

But verification fails.

$ openssl verify -CAfile root-ca.crt server.pem
error 20 at 0 depth lookup:unable to get local issuer certificate

However, if I change the order it seems to work.

$ cat signing-ca.crt subordinate-ca.crt server.crt > server.pem
$ openssl verify -CAfile root-ca.crt server.pem
server.pem: OK

So what would be the error here?

The chain after "cat" looks like below.

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

More info: According to "http://www.herongyang.com/crypto/openssl_verify_2.html", I perform the following test which works.

$ cat signing-ca.crt subordinate-ca.crt > inter.crt
$ openssl verify -CAfile root-ca.crt -untrusted inter.crt server.crt
server.crt: OK

Does that mean all the links are good?

OK, I finally discover that this cannot be done through OpenSSL command line (or at least easily). http://openssl.6102.n7.nabble.com/check-certificate-chain-in-a-pem-file-td43871.html

The original order is in fact backwards. Certs should be followed by the issuing cert until the last cert is issued by a known root per IETF's RFC 5246 Section 7.4.2

This is a sequence (chain) of certificates. The sender's certificate MUST come first in the list. Each following certificate MUST directly certify the one preceding it.

See also SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch for troubleshooting techniques.

But I still don't know why they wrote the spec so that the order matters.

The current top answer is wrong on multiple levels.

First, it misses the main issue, which is simply that 'verify' does not verify chains of certificates in a single file in this way. Proof:

$ cp /etc/ssl/certs/Certigna_Root_CA.pem /tmp/poc.pem
$ echo "unverifiable garbage" >> /tmp/poc.pem
$ openssl verify /tmp/poc.pem 
/tmp/poc.pem: OK

Thus, of the 2 provided orderings in the original question, only the one which has a root certificate at the beginning will report "OK", because that invocation simply checks the root certificate alone, and that is already trusted by the system.

On the subject of ordering certificates, the order as specified in the standard ("The sender's certificate MUST come first in the list.") and that of nginx ("The server certificate must appear before the chained certificates in the combined file") match each other. This corresponds to the OP's first attempt using cat.

The actually correct answer is in the email-thread which was linked by the OP. I'll quote it here:

on the command line, things are unnecessarily difficult.

Here is what you need to do:

1. Split the chain file into one file per certificate, noting the order

2. For each certificate starting with the one above root:

2.1 Concatenate all the previous certificates and the root certificate to one temporary file (This example is for when you are checking the third certifate from the bottom, having already checked cert1.pem and cert2.pem

Unix:    cat cert2.pem cert1.pem root.pem > cert2-chain.pem
Windows: copy /A cert1.pem+cert1.pem+root.pem cert2-chain.pem /A

2.2 Run this command

         openssl verify -CAfile cert2-chain.pem cert3.pem

2.3 If this is OK, proceed to the next one (cert4.pem in this case)

Thus for the first round through the commands would be

Unix: cat root.pem > root-chain.pem Windows: copy /A root.pem root-chain.pem Both: openssl verify -CAfile root-chain.pem cert1.pem

And the second round would be

Unix: cat cert1.pem root.pem > cert1-chain.pem Windows: copy /A cert1.pem+root.pem cert1-chain.pem Both: openssl verify -CAfile cert1-chain.pem cert2.pem

Etc.

Further notes:

1. Step 1 (splitting the file) may be automated like so:

csplit -f cert- $file '/-----BEGIN CERTIFICATE-----/' '{*}'

2. openssl verify takes information about trust from your system (e.g. /etc/ssl/certs/) also, so if you really want to make sure that you're verifying correctly your invocation should be something like openssl verify -verbose -x509_strict -CAfile upto-cert-02 -CAPath nosuchdir cert-01 (where nosuchdir is a non-existing path, and upto-cert-02 is the concatenation of files cert-nn to certt02)