a tiny rant about SSL certificate deployment in web servers

December 7, 2023

did you know that you have to provide a full certificate trust chain for getting almost every web (or any SSL) client to use your website (or service) with TLS?

...but you can still visit some websites with your browser that don't do this?

you probably didn't realize this unless you ran cURL on a website and got a nice little error like this:

(sorry for the people at TUBİTAK ULAKBİM; i had to use a current example)

~ ❯ curl -v https://ulakbim.tubitak.gov.tr
*   Trying 193.140.74.21:443...
* Connected to ulakbim.tubitak.gov.tr (193.140.74.21) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /opt/local/share/curl/curl-ca-bundle.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
~ ❯

but if i use msedge, the site loads just fine with TLS:

Screenshot of ULAKBİM's website, visited with Microsoft Edge.

you might (understandably) think “did the root CA update and not in the trusted keystore?”, but the reality is a little bit different. the latter thought is true, but not in the way you'd expect.

most TLS certificates in the public web would have a chain like this (some exceptions apply, i won't talk about them here):

Chain of Trust

essentially, a cert will point to its issuer until it hits a self-signed one.

many systems will include only the root CAs approved by ~~CA/Browser Forum~~ ^{(post-publication edit, thanks @mmerkel):} the root store operator (usually the vendor) of the system.

this means that the information about intermediates don't exist unless the server gives information about it (the public keys of both the intermediate and root).

when you misconfigure your webserver to use only the host certificate for the public key/certificate, the public key/certificate of the intermediate (and root) will be missing. the only thing available will be the issuer certificate serial and its metadata.

browsers and some OSes like Android include trusted intermediates in their trusted keystore as well. this eliminates the requirement for the server to provide a full trust chain. this obviously create a false sensation of “yeah this works fine”.

but when you check the server with a TLS client like OpenSSL's s_client, the issue comes clear:

(again, sorry for the people at TUBİTAK ULAKBİM)

~ ❯ openssl s_client -connect ulakbim.tubitak.gov.tr:443 -servername ulakbim.tubitak.gov.tr
CONNECTED(00000005)
depth=0 C = TR, ST = KOCAELI, O = TUBITAK BILGEM, CN = *.tubitak.gov.tr
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = TR, ST = KOCAELI, O = TUBITAK BILGEM, CN = *.tubitak.gov.tr
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 C = TR, ST = KOCAELI, O = TUBITAK BILGEM, CN = *.tubitak.gov.tr
verify return:1
---
Certificate chain
 0 s:C = TR, ST = KOCAELI, O = TUBITAK BILGEM, CN = *.tubitak.gov.tr
   i:C = TR, L = Gebze - Kocaeli, O = Turkiye Bilimsel ve Teknolojik Arastirma Kurumu - TUBITAK, OU = Kamu Sertifikasyon Merkezi - Kamu SM, CN = TUBITAK Kamu SM SSL Sertifika Hizmet Saglayicisi - Surum 1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jul 20 13:34:02 2023 GMT; NotAfter: Jul 20 13:34:02 2024 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
[output snipped]

now let's look at a properly configured server:

~ ❯ openssl s_client -connect blog.linuxgemini.space:443 -servername blog.linuxgemini.space
CONNECTED(00000005)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1P5
verify return:1
depth=0 CN = linuxgemini.space
verify return:1
---
Certificate chain
 0 s:CN = linuxgemini.space
   i:C = US, O = Google Trust Services LLC, CN = GTS CA 1P5
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Nov 21 06:59:19 2023 GMT; NotAfter: Feb 19 06:59:18 2024 GMT
 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1P5
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 13 00:00:42 2020 GMT; NotAfter: Sep 30 00:00:42 2027 GMT
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun 19 00:00:42 2020 GMT; NotAfter: Jan 28 00:00:42 2028 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
[output snipped]

you can clearly see that a trust chain is delivered by the server to the client, which the client successfully verifies with its trusted keystore, because GTS Root R1 exists in the keystore. the chain from linuxgemini.space to GTS CA 1P5 to GTS Root R1 is properly documented and delivered by the server.

(now you might ask why GlobalSign Root CA isn't sent, that's because both GlobalSign and Google Trust Services certificates are in the keystore, which makes that part of the chain redundant.)

so in the end; what i want to say is “please use fullchain.pem when you are deploying a server with TLS, you can make one if you don't have any. its just appending each pem file (cert, intermediate cert, root ca cert).”

i had the idea to write this post when i was editing pages on bgp.tools.

Screenshot of a Discord conversation between me and Ben Cartwright-Cox, about this topic while doing stuff with bgp.tools.

sorry again for the infodump, ben.