Client on Debian 9 erroneously reports expired certificate for letsencrypt-issued domain

If I try to access our HTTPS server that has certbot-issued certificate from debian 9, I get the following error:

 # curl -v https://hu.dbpedia.org/
 *   Trying 195.111.2.82...
 * TCP_NODELAY set
 * Connected to hu.dbpedia.org (195.111.2.82) port 443      (#0)
 * ALPN, offering h2
 * ALPN, offering http/1.1
 * Cipher selection:      ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
 * successfully set certificate verify locations:
 *   CAfile: /etc/ssl/certs/ca-certificates.crt
   CApath: /etc/ssl/certs
 * TLSv1.2 (OUT), TLS header, Certificate Status (22):
 * TLSv1.2 (OUT), TLS handshake, Client hello (1):
 * TLSv1.2 (IN), TLS handshake, Server hello (2):
 * TLSv1.2 (IN), TLS handshake, Certificate (11):
 * TLSv1.2 (OUT), TLS alert, Server hello (2):
 * SSL certificate problem: certificate has expired
 * Curl_http_done: called premature == 1
 * stopped the pause stream!
 * Closing connection 0
 curl: (60) SSL certificate problem: certificate has expired

However, if I try the same command from debian 10, it succeeds.

I tried to simply copy all ca-certificates from a debian 10 VM to the debian 9 VM (to /usr/local/share/ca-certificates) with rsync, and then ran update-ca-certificates which seemingly added 400+ certificates. Unfortunately, it did not help. This is no wonder, as it seems that there are the same certificates on both debian 9 and 10, apparently.

My question is: How can I access sites with certbot certificates from debian 9 machines without ignoring certificate verification altogether

First off, Debian 9 is EOL. But as the clients may not be under your control, you may of course want to try to cater to them in this breakage.

I assume that while the question only mentions certbot, that it's really specifically about Letsencrypt.
(The tool certbot itself is an ACME protocol client which is also used with other ACME-based CAs, so there is some room for confusion here.)

The problem at hand would appear to be the combination of:

• The old Letsencrypt root ("DST Root CA X3") has expired
• The new default LE chain tries to be "extra compatible" by presenting an optional extension of the chain where the new root ("ISRG Root X1") is presented as a cross-signed intermediate for the old root (as very old Android versions still accept the expired root, but do not have the new root)
• Openssl 1.0 has a bug causing it to just try the first chain it sees and if it doesn't like that, it doesn't look at any other possibilities (ie, the shorter new chain ending at X1 vs the longer "compatibility" extension of that chain going through X1 to X3).
• libcurl3 on Debian 9 is linked to libssl 1.0

If you instead present the new LE certificate chain that does not try to be extra compatible, just ending at the new root (X1), it allows libssl 1.0 to work (but then you lose compatibility with really old Android instead).

Other than that, other CAs (ACME or otherwise) is probably the option to consider.

Warning! Please plan OS upgrade path. The below advice should be applied only in emergency situation to quickly fix a critical system.

Below solution works on Debian Jessy 8 and should on Stretch 9 missing updates too. I just tested it on docker run -it debian:jessie bash, apt-get update && apt-get install curl.

Before:

# curl -I https://hu.dbpedia.org
curl: (60) SSL certificate problem: certificate has expired
More details here: http://curl.haxx.se/docs/sslcerts.html
....

First check if you have offending DST Root CA X3 cert present:

# grep X3 /etc/ca-certificates.conf 
mozilla/DST_Root_CA_X3.crt

Old Debian releases have the proper ISRG Root X1 present too:

# grep X1 /etc/ca-certificates.conf 
mozilla/ISRG_Root_X1.crt

This is going to disable X3:

# sed -i '/^mozilla\/DST_Root_CA_X3/s/^/!/' /etc/ca-certificates.conf && update-ca-certificates -f

Your domain responds to curl fine:

# curl -I https://hu.dbpedia.org/
HTTP/1.1 200 OK
...

Again, plan an upgrade please.

DST Root CA X3 root certificate expired on Sep 30 14:01:15 2021 GMT. It was used as one of certification paths for Let’s Encrypt certificates Older cURL version has a bug that will cause expired root to fail connection instead of trying other roots in local ca store.

As of 30 september 2021 ca store for curl (https://curl.se/docs/caextract.html) still contains expired DST Root CA X3 root certificate so updating it won't fix problem. You can either update your cURL (which might be quite challenging in some situations) or edit local ca store (f.e. /etc/pki/tls/certs/ca-bundle.crt) and manually remove certificate after line "DST Root CA X3"

I've just had this problem with a Docker image based on Debian Stretch. The solution has been easy, I suppose Debian has patched libssl1.0.2 to fix the chain selection issue Håkan has mentioned. In my case, updating just the certificates and libssl1.0.2 by running apt install -y libssl1.0.2 ca-certificates (with apt update for containers beforehand) has been enough.

For Debian 8 and 9, I just do that for updating cert on host and no longer have the error 60: SSL certificate problem: certificate has expired : https://github.com/xenetis/letsencrypt-expiration

Just run, it should work :

wget -O - https://raw.githubusercontent.com/xenetis/letsencrypt-expiration/main/letsencrypt-expiration.sh | bash