How to fix certificate chain with letsencrypt / certbot?

Solution 1:

Try openssl s_client and let you show the certs. The command is:

$ openssl s_client -connect co2avatar.org:443 -servername co2avatar.org -showcerts

You will find that your server returns a certificate for CN = gitlab.sustainable-data-platform.org and a subject alternative name which includes your domain DNS:co2-avatar.com. So the certificate itself is fine.

If you want to combine everything into one pipeline of commands to see the content of your certificate:

echo | openssl s_client -connect co2avatar.org:443 -servername co2avatar.org -showcerts 2>/dev/null |sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | openssl x509 -noout -text

Whats missing is the intermediate certificate. This should be sent by the server as well, but the first command shows you that it is not there - only the certificate is sent by your server.

So the failing openssl is correct, as indeed the intermediate certificate is missing.

So to solve it, you need to tweak your apache configuration. This is how your configuration could look like:

Filename should be similar to /etc/apache2/sites-enabled/co2-avatar.com-le-ssl.conf

<IfModule mod_ssl.c>
SSLStaplingCache shmcb:/var/run/apache2/stapling_cache(128000)
<VirtualHost *:443>
        ServerName co2-avatar.com
        ServerAlias www.co2-avatar.com
#... 
#... insert your other stuff here...
#...

SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/co2-avatar.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
SSLUseStapling on
</VirtualHost>
</IfModule>

Based on your description, my best guess is that the following line is wrong in your config: SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/cert.pem. It should be replaced with SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem, in order to send also the intermediate(s).

Solution UPDATE (after discussion)

It turned out in discussion that the openssl and Apache version used on this CentOS server is just older, so some features are unsupported. (Apache 2.4.6, OpenSSL 1.0.2k, intermediate configuration, no HSTS, no OCSP)

According to Mozilla SSL Configuration Generator the following generic configuration could be used in this case:

<VirtualHost *:443>
    SSLEngine on
    SSLCertificateFile      /path/to/signed_certificate
    SSLCertificateChainFile /path/to/intermediate_certificate
    SSLCertificateKeyFile   /path/to/private_key
</VirtualHost>

Translated to this specific case, a resulting working config would be as the following:

<VirtualHost *:443>
    ServerName  sustainable-data-platform.org
    ServerAlias co2-avatar.com
    ServerAlias ... <include all other SAN names here>
    
    SSLEngine on
    SSLCertificateFile      /etc/letsencrypt/live/co2-avatar.com/cert.pem
    SSLCertificateChainFile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem
    SSLCertificateKeyFile   /etc/letsencrypt/live/co2-avatar.com/privkey.pem

</VirtualHost>

As note for such old installations

Cross-Signed Let’s Encrypt R3 and DST Root CA X3, intermediate and root certificates will expire on Sep 29, 2021 and Sep 30, 2021 respectively. So since May 4, 2021, The newly issued certificates use a longer chain with cross-signed ISRG Root X1 as an intermediate certificate.

Unfortunately, due to the way certificate paths are built and verified, not all implementations of TLS can successfully verify the cross-sign. This is the case with OpenSSL 1.0.2. Hence, programs running on RHEL/CentOS 7 that use OpenSSL will likely fail to verify the new certificate chain or establish TLS connection. Upgrading to newer Openssl versions on such platforms is not straightforward.

There are a few options: either update the trust store (remove DST Root CA X3 root certificate - once it is removed, impact should be minimal) on the client side (or) change the certificate chain on the server side.

For Nginx

For Nginx there is only one parameter to specify the cert file. You should use the fullchain.pem provided by certbot to get it working correctly.

The right configuration in the server block for the given virtualhost would be as follow:

server {
  ...
  ssl_certificate /etc/letsencrypt/live/co2-avatar.com/fullchain.pem;  -> replaced cert.pem for fullchain.pem
  ssl_certificate_key /etc/letsencrypt/live/co2-avatar.com/privkey.pem;
}

References

• DST Root CA X3 Expiration (September 2021)
• Let’s Encrypt change affects OpenSSL 1.0.x and CentOS 7