SSL Root CA certificate is not recognized, although present in the trust store. Why?

Background:

• Ubuntu Server 14.10 64-bit on aws.amazon.com/ec2
• Cheap PositiveSSL server certificate from COMODO
• 1 server certificate, 2 intermediate CA certificates and 1 Root CA certificate as ZIP archive from COMODO
• Citadel's WebCit httpsd

Problem:

The concatenated certificate chain seems to be correct but verification fails.

openssl s_client myhost:port

shows the certificate chain and the issuer-subject pairs line up correctly through the chain, but:

verify error:num=19:self signed certificate in certificate chain

The root CA certificate is not accepted by openssl, although it is found per default in the Ubuntu server trust store.

Specifically: AddTrustExternalCARoot.crt received per email from COMODO and /etc/ssl/certs/AddTrust_External_Root.pem which links to /usr/share/ca-certificates/mozilla/AddTrust_External_Root.crt are indentical.

What is wrong here?

OpenSSL at least through current (1.0.2a) has a bug where s_client with NO -CA{path,file} argument doesn't actually use the default truststore as it should, and thus fails to verify certs that are valid according to that truststore. (Also s_server and s_time, but caring about verification in those is rare.) See https://serverfault.com/questions/607233/how-to-make-openssl-s-client-using-default-ca . A fix is announced in dev, but may take some time to be released and distributed. In the meantime you need to explicitly specify the -CA* argument(s). Note that openssl verify does not have this bug, and therefore correctly reported the cert/chain as valid.

UPDATES 2015/08/26: fix was released 2015/06/12 in 1.0.1o and 1.0.2c. Also, while investigating something else I found that RedHat packages may have been okay. More specifically the CentOS source RPM for openssl-1.0.1e-30.el6.11 which I understand is a copy of the RedHat one (but can't easily confirm) contains openssl-1.0.1c-default-paths.patch which contains changes to s_client.c s_server.c s_time.c dated 2012/12/06 that appear equivalent to (though not textually the same as) the 2015/06/12 upstream fixes. Assuming this patch was applied in RedHat and CentOS packages, which I can't easily go back and check, they would (have) work(ed) as expected.

I faced a similar issue with Comodo certificates recently when developing a script using Ruby. In the end it was that OpenSSL did not have it in the store, even though it looked like it did.

To test this, download all of the Comodo intermediate certs and create a cert bundle something like this (you'll need to use different cert names depending on what you downloaded):

cat EssentialSSLCA_2.crt ComodoUTNSGCCA.crt UTNAddTrustSGCCA.crt AddTrustExternalCARoot.crt > yourDomain.ca-bundle

Comodo has an article on how to do this.

Once done, try verifying the certificate again using OpenSSL and specifying the cert store on the command line:

openssl verify -untrusted yourDomain.ca-bundle cert.pem

That example was adapted from this Unix and Linux StackExchange article.

Once you've determined which certificate it is, it should be possible to add the certificate to the local cert store, which is detailed here for Ubuntu, and is something like:

Create a directory for extra CA certificates in /usr/share/ca-certificates

sudo mkdir /usr/share/ca-certificates/extra

Copy the '.crt' file to the directory

sudo cp foo.crt /usr/share/ca-certificates/extra/foo.crt

Let Ubuntu add the '.crt' file's path relative to /usr/share/ca-certificates to /etc/ca-certificates.conf

sudo dpkg-reconfigure ca-certificates