Apache seems to be using old expired certificate even though new one is installed

Apache 2.2.3/mod_ssl/CentOS 5.5 VPS

Our certificate expired on 2011-10-06, and even though we have seemingly installed the new one correctly, browsing to the site still shows an expired certificate! I've tried deleting my browser cache and using several different browsers. Relevant lines from the ssl.conf file (I've excluded those commented out.):

Listen 127.0.0.1:443
SSLSessionCache         shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout  300
# Note - I tried disabling SSLSessionCache with the "none" setting but it didn't help.
<VirtualHost 127.0.0.1:443>
SSLEngine on
SSLProtocol all -SSLv2
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
SSLCertificateFile /var/certs/gentlemanjoe.com/new2011/gentlemanjoe.com.crt
SSLCertificateKeyFile /var/certs/gentlemanjoe.com/new2011/gentlemanjoe.com.key
SSLCertificateChainFile /var/certs/gentlemanjoe.com/new2011/gd_bundle.crt
SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
CustomLog logs/ssl_request_log \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
ServerAdmin [email protected]
DocumentRoot /var/www/gentlemanjoe.com
ServerName gentlemanjoe.com
<Directory /var/www/gentlemanjoe.com>
    AllowOverride All
    Order deny,allow
    allow from all
</Directory>          
</VirtualHost>

Things I've Checked

First I tried moving the old cert and key files to a completely different folder to make sure Apache wasn't still grabbing them somehow. Nothing changed. For fun I tried temporarily renaming the new cert and key files, and Apache dutifully complained and refused to start.

Then I tried to make sure I wasn't being fooled by editing the wrong config file. Using "locate" I found only one httpd.conf file under /etc/httpd/conf/httpd.conf. I also used "locate" to verify that there is only one ssl.conf file, /etc/httpd/conf.d/ssl.conf. The key file is what I generated using OpenSSL, following the instructions GoDaddy gave for generating the CSR.

I have verified that I am working with the right site by uploading a test.html file to the /var/www/gentlemanjoe.com folder and verifying that I can browse to it. But if I try to view the test file in HTTPS, I get the same certificate expiration warning.

I verified that the cert itself has the right expiration date:

openssl x509 -in /var/certs/gentlemanjoe.com/new2011/gentlemanjoe.com.crt -noout -text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            07:e7:49:69:97:96:16
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., OU=http://certificates.godaddy.com/repository, CN=Go Daddy Secure Certification Authority/serialNumber=07969287
        Validity
            Not Before: Oct 21 17:37:55 2011 GMT
            Not After : Oct  8 21:16:03 2013 GMT
        Subject: C=CA, ST=BC, L=Burnaby, O=Diamond Bailey Consolidated Commercial Services Ltd, OU= , CN=www.gentlemanjoe.com

I tried re-keying the cert at GoDaddy with a new CSR and everything seems to work, but I get the same result in the browser.

Possible Clue #1

Whenever I do "apachectl restart", I see this in the error_log file:

[Fri Oct 21 18:03:33 2011] [notice] SIGHUP received.  Attempting to restart
[Fri Oct 21 18:03:33 2011] [notice] Digest: generating secret for digest authentication ...
[Fri Oct 21 18:03:33 2011] [notice] Digest: done
[Fri Oct 21 18:03:33 2011] [info] APR LDAP: Built with OpenLDAP LDAP SDK
[Fri Oct 21 18:03:33 2011] [info] LDAP: SSL support available
[Fri Oct 21 18:03:33 2011] [info] Init: Seeding PRNG with 256 bytes of entropy
[Fri Oct 21 18:03:33 2011] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Fri Oct 21 18:03:33 2011] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Fri Oct 21 18:03:33 2011] [info] Shared memory session cache initialised
[Fri Oct 21 18:03:33 2011] [info] Init: Initializing (virtual) servers for SSL
[Fri Oct 21 18:03:33 2011] [warn] RSA server certificate CommonName (CN) `www.gentlemanjoe.com' does NOT match server name!?
[Fri Oct 21 18:03:33 2011] [info] Server: Apache/2.2.3, Interface: mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5
[Fri Oct 21 18:03:34 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Fri Oct 21 18:03:34 2011] [info] Server built: Aug 30 2010 12:28:40

The GoDaddy techs tell me that the www vs non-www should not matter, and I tend to agree, since the security warning in my browser is not complaining about a server name mismatch, but rather an expiration, indicating that the old certificate is still being loaded somehow.

Possible Clue #2

The HTTP Server response header for http://gentlemanjoe.com says "Andromeda" rather than "Apache". This seems weird to me since my Googling of "Andromeda" turns up a media-server type project, which wouldn't be installed on this server (but I can't say that with certainty since I didn't set any of this up, the usual admin/developer is on holiday and I'm just helping a friend with his site.) Also, the httpd.conf file does not contain the string "Andromeda" indicating that it hasn't been modified to spit this out. So it might be the Magento e-commerce platform he's using, but what would be the point of replacing the standard Apache response header?

Something's in front of Apache. Check out that config:

Listen 127.0.0.1:443
....
<VirtualHost 127.0.0.1:443>

It's listening on localhost only, so internet clients aren't hitting this service directly - they're likely getting proxied.

For the sanity check that Apache's loading the right cert, hit the service directly on Apache's listener: openssl s_client -connect 127.0.0.1:443 -showcerts

Not sure about the Andromeda header, so, let's find the process: lsof -i.

Apache will have 127.0.0.1:443, while some other service has 0.0.0.0:443 (or the VPS's public address :443) - that's the one that needs the new cert.

A common source of this problem is multiple running instances of Apache. The config changes are picked up by a process that you (re)start but the request is served by an old process which is running with old configuration.

Stop the service:

service apache2 stop

Check if the site is still accessible. If yes, then you have identified the cause.

Now run

ps aux | grep apache

It will give you list of running apache2 process and their PIDs. Kill them all (Note, this command may also return unrelated processes with Apache in their name/user etc. like Apache Tomcat, you might not want kill them.)

kill <pid>

Run ps aux again and ensure that processes are no longer running.

Check again if site is accessible. It shouldn't be.

Now start apache service

service apache2 start

Verify that the new certificate is being served.

If you don't want to kill processes, you may reboot the system. It will have the same effect.