Can't get Nginx to serve correct certificate chain
I'm trying to set up Nginx (1.4.6-1ubuntu3.1) with a StartSSL certificate. I've been following the documentation to get it working, but Nginx only serves the server certificate, not the intermediate.
My server config:
server {
server_name lanzz.org www.lanzz.org;
root /var/www/lanzz.org/public;
index index.html;
include listen.conf;
ssl_certificate /etc/ssl/nginx/lanzz.org.chained.pem;
ssl_certificate_key /etc/ssl/nginx/lanzz.org.key;
}
My certificate bundle (/etc/ssl/nginx/lanzz.org.chained.pem):
-----BEGIN CERTIFICATE-----
MIIGMzCCBRugAwIBAgIDE5WyMA0GCSqGSIb3DQEBCwUAMIGMMQswCQYDVQQGEwJJ
TDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0
YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20gQ2xhc3Mg
MSBQcmltYXJ5IEludGVybWVkaWF0ZSBTZXJ2ZXIgQ0EwHhcNMTQxMDE0MTMzNjI1
WhcNMTUxMDE1MTUzODI3WjBKMQswCQYDVQQGEwJHQjEWMBQGA1UEAxMNd3d3Lmxh
bnp6Lm9yZzEjMCEGCSqGSIb3DQEJARYUcG9zdG1hc3RlckBsYW56ei5vcmcwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCs8AMEaHFRZ4S3gGYhDJg05vj3
EadGIi29EJf5iW1YwKduuA62Zv9D2JGh2FCBJDBTswefxFbs2v/HHdP70gr0l669
Vz47RUJgn6xH13xEnVv5btQfPtioLQJNwLnBDR3ycw+9I/CGq+/BmStXBT2fTlDp
7FlDaemkc/mjd4TM6DBL0mfsAfqcSA4GHgQraJSwMyRGn3lon02mOWsDso6nTMEt
QYmCvYoM7wVtiBxKGP9Q6Nz3s5Ouc1U7mxKxuLNIO5ZeT+zocW7HQXk1sGal/Hxi
Y+Us/SsmcDAvqvI9f44Xe4StMfPDBphEDrOvJt9/zuDu8SNMnEA1cqZHGQFVAgMB
AAGjggLdMIIC2TAJBgNVHRMEAjAAMAsGA1UdDwQEAwIDqDATBgNVHSUEDDAKBggr
BgEFBQcDATAdBgNVHQ4EFgQUiw5c8ahxug4Ltshgeob3+CjaFSAwHwYDVR0jBBgw
FoAU60I00Jiwq5/0G2sI98xkLu8OLEUwIwYDVR0RBBwwGoINd3d3Lmxhbnp6Lm9y
Z4IJbGFuenoub3JnMIIBVgYDVR0gBIIBTTCCAUkwCAYGZ4EMAQIBMIIBOwYLKwYB
BAGBtTcBAgMwggEqMC4GCCsGAQUFBwIBFiJodHRwOi8vd3d3LnN0YXJ0c3NsLmNv
bS9wb2xpY3kucGRmMIH3BggrBgEFBQcCAjCB6jAnFiBTdGFydENvbSBDZXJ0aWZp
Y2F0aW9uIEF1dGhvcml0eTADAgEBGoG+VGhpcyBjZXJ0aWZpY2F0ZSB3YXMgaXNz
dWVkIGFjY29yZGluZyB0byB0aGUgQ2xhc3MgMSBWYWxpZGF0aW9uIHJlcXVpcmVt
ZW50cyBvZiB0aGUgU3RhcnRDb20gQ0EgcG9saWN5LCByZWxpYW5jZSBvbmx5IGZv
ciB0aGUgaW50ZW5kZWQgcHVycG9zZSBpbiBjb21wbGlhbmNlIG9mIHRoZSByZWx5
aW5nIHBhcnR5IG9ibGlnYXRpb25zLjA1BgNVHR8ELjAsMCqgKKAmhiRodHRwOi8v
Y3JsLnN0YXJ0c3NsLmNvbS9jcnQxLWNybC5jcmwwgY4GCCsGAQUFBwEBBIGBMH8w
OQYIKwYBBQUHMAGGLWh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9zdWIvY2xhc3Mx
L3NlcnZlci9jYTBCBggrBgEFBQcwAoY2aHR0cDovL2FpYS5zdGFydHNzbC5jb20v
Y2VydHMvc3ViLmNsYXNzMS5zZXJ2ZXIuY2EuY3J0MCMGA1UdEgQcMBqGGGh0dHA6
Ly93d3cuc3RhcnRzc2wuY29tLzANBgkqhkiG9w0BAQsFAAOCAQEAEom5lVxCbfu9
3K+BuowfCgTyA4keiQcYmTUJYXRBV9OiFUc/V5tXyhmgdyYeJB3oKMaEQ3glClZm
ueXUkALhaIlEzXjoNZgOh/bdbBPwfOq2WMBaWJbXX3x4C77s52zPBbkqhsBq5nge
1YDho1Z7tVYe8iyqBPUIFq0//LfGVAMoR7ZSwVpUgeiWs3oVKQMyR2BzrNSq392L
f8kIdIQ8fTgomtCZ2F2qp8EBIxl6G4UVpeN3Nes9UODpdNeG6lzLoSvYsVX+eWXx
3m29Aem42vlMhtQL1dtahb71nTY26qWAG337fwQoRxAXht3vSs0HhdUEHyGQGsuu
k+fcnm/wiQ==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIGNDCCBBygAwIBAgIBGDANBgkqhkiG9w0BAQUFADB9MQswCQYDVQQGEwJJTDEW
MBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwg
Q2VydGlmaWNhdGUgU2lnbmluZzEpMCcGA1UEAxMgU3RhcnRDb20gQ2VydGlmaWNh
dGlvbiBBdXRob3JpdHkwHhcNMDcxMDI0MjA1NDE3WhcNMTcxMDI0MjA1NDE3WjCB
jDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzApBgNVBAsT
IlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcxODA2BgNVBAMTL1N0
YXJ0Q29tIENsYXNzIDEgUHJpbWFyeSBJbnRlcm1lZGlhdGUgU2VydmVyIENBMIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtonGrO8JUngHrJJj0PREGBiE
gFYfka7hh/oyULTTRwbw5gdfcA4Q9x3AzhA2NIVaD5Ksg8asWFI/ujjo/OenJOJA
pgh2wJJuniptTT9uYSAK21ne0n1jsz5G/vohURjXzTCm7QduO3CHtPn66+6CPAVv
kvek3AowHpNz/gfK11+AnSJYUq4G2ouHI2mw5CrY6oPSvfNx23BaKA+vWjhwRRI/
ME3NO68X5Q/LoKldSKqxYVDLNM08XMML6BDAjJvwAwNi/rJsPnIO7hxDKslIDlc5
xDEhyBDBLIf+VJVSH1I8MRKbf+fAoKVZ1eKPPvDVqOHXcDGpxLPPr21TLwb0pwID
AQABo4IBrTCCAakwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYD
VR0OBBYEFOtCNNCYsKuf9BtrCPfMZC7vDixFMB8GA1UdIwQYMBaAFE4L7xqkQFul
F2mHMMo0aEPQQa7yMGYGCCsGAQUFBwEBBFowWDAnBggrBgEFBQcwAYYbaHR0cDov
L29jc3Auc3RhcnRzc2wuY29tL2NhMC0GCCsGAQUFBzAChiFodHRwOi8vd3d3LnN0
YXJ0c3NsLmNvbS9zZnNjYS5jcnQwWwYDVR0fBFQwUjAnoCWgI4YhaHR0cDovL3d3
dy5zdGFydHNzbC5jb20vc2ZzY2EuY3JsMCegJaAjhiFodHRwOi8vY3JsLnN0YXJ0
c3NsLmNvbS9zZnNjYS5jcmwwgYAGA1UdIAR5MHcwdQYLKwYBBAGBtTcBAgEwZjAu
BggrBgEFBQcCARYiaHR0cDovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5LnBkZjA0
BggrBgEFBQcCARYoaHR0cDovL3d3dy5zdGFydHNzbC5jb20vaW50ZXJtZWRpYXRl
LnBkZjANBgkqhkiG9w0BAQUFAAOCAgEAIQlJPqWIbuALi0jaMU2P91ZXouHTYlfp
tVbzhUV1O+VQHwSL5qBaPucAroXQ+/8gA2TLrQLhxpFy+KNN1t7ozD+hiqLjfDen
xk+PNdb01m4Ge90h2c9W/8swIkn+iQTzheWq8ecf6HWQTd35RvdCNPdFWAwRDYSw
xtpdPvkBnufh2lWVvnQce/xNFE+sflVHfXv0pQ1JHpXo9xLBzP92piVH0PN1Nb6X
t1gW66pceG/sUzCv6gRNzKkC4/C2BBL2MLERPZBOVmTX3DxDX3M570uvh+v2/miI
RHLq0gfGabDBoYvvF0nXYbFFSF87ICHpW7LM9NfpMfULFWE7epTj69m8f5SuauNi
YpaoZHy4h/OZMn6SolK+u/hlz8nyMPyLwcKmltdfieFcNID1j0cHL7SRv7Gifl9L
WtBbnySGBVFaaQNlQ0lxxeBvlDRr9hvYqbBMflPrj0jfyjO1SPo2ShpTpjMM0InN
SRXNiTE8kMBy12VLUjWKRhFEuT2OKGWmPnmeXAhEKa2wNREuIU640ucQPl2Eg7PD
wuTSxv0JS3QJ3fGz0xk+gA2iCxnwOOfFwq/iI9th4p1cbiCJSS4jarJiwUW0n6+L
p/EiO/h94pDQehn7Skzj0n1fSoMD7SfWI55rjbRZotnvbIIp3XUZPD9MEI3vu3Un
0q6Dp6jOW6c=
-----END CERTIFICATE-----
Output from openssl s_client -connect lanzz.org:443:
CONNECTED(00000003)
depth=0 C = GB, CN = www.lanzz.org, emailAddress = [email protected]
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = GB, CN = www.lanzz.org, emailAddress = [email protected]
verify error:num=27:certificate not trusted
verify return:1
depth=0 C = GB, CN = www.lanzz.org, emailAddress = [email protected]
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/C=GB/CN=www.lanzz.org/[email protected]
i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGMzCCBRugAwIBAgIDE5WyMA0GCSqGSIb3DQEBCwUAMIGMMQswCQYDVQQGEwJJ
TDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0
YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20gQ2xhc3Mg
MSBQcmltYXJ5IEludGVybWVkaWF0ZSBTZXJ2ZXIgQ0EwHhcNMTQxMDE0MTMzNjI1
WhcNMTUxMDE1MTUzODI3WjBKMQswCQYDVQQGEwJHQjEWMBQGA1UEAxMNd3d3Lmxh
bnp6Lm9yZzEjMCEGCSqGSIb3DQEJARYUcG9zdG1hc3RlckBsYW56ei5vcmcwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCs8AMEaHFRZ4S3gGYhDJg05vj3
EadGIi29EJf5iW1YwKduuA62Zv9D2JGh2FCBJDBTswefxFbs2v/HHdP70gr0l669
Vz47RUJgn6xH13xEnVv5btQfPtioLQJNwLnBDR3ycw+9I/CGq+/BmStXBT2fTlDp
7FlDaemkc/mjd4TM6DBL0mfsAfqcSA4GHgQraJSwMyRGn3lon02mOWsDso6nTMEt
QYmCvYoM7wVtiBxKGP9Q6Nz3s5Ouc1U7mxKxuLNIO5ZeT+zocW7HQXk1sGal/Hxi
Y+Us/SsmcDAvqvI9f44Xe4StMfPDBphEDrOvJt9/zuDu8SNMnEA1cqZHGQFVAgMB
AAGjggLdMIIC2TAJBgNVHRMEAjAAMAsGA1UdDwQEAwIDqDATBgNVHSUEDDAKBggr
BgEFBQcDATAdBgNVHQ4EFgQUiw5c8ahxug4Ltshgeob3+CjaFSAwHwYDVR0jBBgw
FoAU60I00Jiwq5/0G2sI98xkLu8OLEUwIwYDVR0RBBwwGoINd3d3Lmxhbnp6Lm9y
Z4IJbGFuenoub3JnMIIBVgYDVR0gBIIBTTCCAUkwCAYGZ4EMAQIBMIIBOwYLKwYB
BAGBtTcBAgMwggEqMC4GCCsGAQUFBwIBFiJodHRwOi8vd3d3LnN0YXJ0c3NsLmNv
bS9wb2xpY3kucGRmMIH3BggrBgEFBQcCAjCB6jAnFiBTdGFydENvbSBDZXJ0aWZp
Y2F0aW9uIEF1dGhvcml0eTADAgEBGoG+VGhpcyBjZXJ0aWZpY2F0ZSB3YXMgaXNz
dWVkIGFjY29yZGluZyB0byB0aGUgQ2xhc3MgMSBWYWxpZGF0aW9uIHJlcXVpcmVt
ZW50cyBvZiB0aGUgU3RhcnRDb20gQ0EgcG9saWN5LCByZWxpYW5jZSBvbmx5IGZv
ciB0aGUgaW50ZW5kZWQgcHVycG9zZSBpbiBjb21wbGlhbmNlIG9mIHRoZSByZWx5
aW5nIHBhcnR5IG9ibGlnYXRpb25zLjA1BgNVHR8ELjAsMCqgKKAmhiRodHRwOi8v
Y3JsLnN0YXJ0c3NsLmNvbS9jcnQxLWNybC5jcmwwgY4GCCsGAQUFBwEBBIGBMH8w
OQYIKwYBBQUHMAGGLWh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9zdWIvY2xhc3Mx
L3NlcnZlci9jYTBCBggrBgEFBQcwAoY2aHR0cDovL2FpYS5zdGFydHNzbC5jb20v
Y2VydHMvc3ViLmNsYXNzMS5zZXJ2ZXIuY2EuY3J0MCMGA1UdEgQcMBqGGGh0dHA6
Ly93d3cuc3RhcnRzc2wuY29tLzANBgkqhkiG9w0BAQsFAAOCAQEAEom5lVxCbfu9
3K+BuowfCgTyA4keiQcYmTUJYXRBV9OiFUc/V5tXyhmgdyYeJB3oKMaEQ3glClZm
ueXUkALhaIlEzXjoNZgOh/bdbBPwfOq2WMBaWJbXX3x4C77s52zPBbkqhsBq5nge
1YDho1Z7tVYe8iyqBPUIFq0//LfGVAMoR7ZSwVpUgeiWs3oVKQMyR2BzrNSq392L
f8kIdIQ8fTgomtCZ2F2qp8EBIxl6G4UVpeN3Nes9UODpdNeG6lzLoSvYsVX+eWXx
3m29Aem42vlMhtQL1dtahb71nTY26qWAG337fwQoRxAXht3vSs0HhdUEHyGQGsuu
k+fcnm/wiQ==
-----END CERTIFICATE-----
subject=/C=GB/CN=www.lanzz.org/[email protected]
issuer=/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
---
No client certificate CA names sent
---
SSL handshake has read 2256 bytes and written 451 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : ECDHE-RSA-AES256-SHA
Session-ID: 958F818087118CA5F3344796B13FD463097B08ACF4F5497A3AD7A40D550FA155
Session-ID-ctx:
Master-Key: 3AB94A204BF8E30BEDC615832E7DD5B2347458A6324A85C8C02BD441AF9F39BA09191BEB85A5F40ED99049013A51F7AB
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 27 5f 93 a2 81 ed c3 e7-ba 41 27 0f 8f 95 be f5 '_.......A'.....
0010 - 68 a6 7a 8b 1a 84 71 6a-00 cd ec 94 fc 3f 2c 64 h.z...qj.....?,d
0020 - d8 b1 19 76 ba 8e ff e6-97 39 89 fb 1d 69 47 4a ...v.....9...iGJ
0030 - b3 40 96 a5 30 9e c6 b6-79 7f 93 36 02 31 4d 16 [email protected].
0040 - 2b d9 d6 0b 3c a9 07 10-63 af 89 b2 19 0e ac b3 +...<...c.......
0050 - 3f c2 41 df f5 23 62 5c-11 d8 f6 be 55 1e e2 74 ?.A..#b\....U..t
0060 - 6c 4f cd ba 72 e1 a1 f7-b4 fb d1 19 6d 33 d5 63 lO..r.......m3.c
0070 - d6 94 a0 e6 e5 0c 33 e4-67 e3 49 fe fc b1 d5 25 ......3.g.I....%
0080 - 9d d7 d3 72 b4 fb 5a 1f-6e 7f 5e 4b 41 21 3a d5 ...r..Z.n.^KA!:.
0090 - 37 ef 4a b4 35 2d 14 bd-29 77 6d c5 6c c5 3c a4 7.J.5-..)wm.l.<.
Start Time: 1422525959
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
---
As you can see, only one certificate is present in the chain, the intermediate is not served by Nginx.
I've verified that Nginx is seeing the correct bundle, because if I swap the server cert with the intermediate cert I do see the expected SSL_CTX_use_PrivateKey_file error mentioned in the docs. When I put them in the correct order I see no errors at startup, no errors at request time, but no intermediate certificate in the SSL session.
The chain is otherwise correct, as I have verified that a browser that already has the intermediate will accept the server certificate, and I've verified that the fingerprints for the intermediate and the server certificates I see in the browser match what I get for the certificates in my bundle on the server.
I'm at a complete loss here, and I don't know what else to try. I think everything is set up by the book, but I still don't get the expected result in s_client. I could not find anyone else even having such a problem, much less any existing solution to it. I've even tried including the root certificate in the bundle, which had no effect whatsoever. Any tips or pointers will be much appreciated.
Solution 1:
Turns out, the Nginx documentation is somewhat misleading and goes at odds with SNI.
Running openssl s_client -connect lanzz.org:443:
CONNECTED(00000003)
depth=0 C = GB, CN = www.lanzz.org, emailAddress = [email protected]
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = GB, CN = www.lanzz.org, emailAddress = [email protected]
verify error:num=27:certificate not trusted
verify return:1
depth=0 C = GB, CN = www.lanzz.org, emailAddress = [email protected]
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/C=GB/CN=www.lanzz.org/[email protected]
i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
---
Running openssl s_client -connect lanzz.org:443 -servername lanzz.org:
CONNECTED(00000003)
depth=1 C = IL, O = StartCom Ltd., OU = Secure Digital Certificate Signing, CN = StartCom Class 1 Primary Intermediate Server CA
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
0 s:/C=GB/CN=www.lanzz.org/[email protected]
i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
1 s:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority
---
Apparently, on one hand I had a default virtual host configured with the wrong certificate file (one without the intermediate), and on the other openssl s_client does not automatically provide a SNI extension for the hostname passed to the -connect argument. The Nginx documentation naively provides a command that would only test the SSL setup of the default virtual host, which I would assume is rarely what people usually want.
TL;DR: Supply the -servername parameter when testing your SSL chains:
openssl s_client -connect <hostname>:443 -servername <hostname>