Missing TLS intermediate certificates can create mysterious browser problems

Today I wound up helping someone with a weird and mysterious browser problem, where they couldn't connect to a particular HTTPS site with either Chrome and Microsoft Edge on their up to date Windows machine; when they attempted to do so, Chrome (and Edge) reported that the TLS certificate was issued by an unknown Certificate Authority. Firefox on their machine could connect and I could connect with various browsers (including from Chrome and Microsoft Edge on some Windows 10 machines I have access to). Given the title of this entry, you already know the ultimate cause; the website's TLS certificate was signed by an intermediate certificate that the website wasn't serving.

With a missing intermediate certificate, the website's TLS certificate looked like it was signed by an unknown CA, that being the "CA" of the name of the intermediate certificate (although neither Chrome nor Microsoft Edge reported the name of the unknown CA, probably for good reasons). The website worked in other browsers because browsers silently cache intermediate TLS certificates that they've seen before, in addition to being willing to download them under the right circumstances. The person who had this problem almost never uses Chrome or Microsoft Edge (they mostly use Firefox), so those browsers had never had the chance to see this particular intermediate certificate before.

(To make things worse, I believe that this particular intermediate certificate is deprecated, so you won't find many sites still using it and providing it to you.)

This is a hard problem for a lot of people to even see, much less identify. The browser's intermediate certificate cache is basically invisible and I think most browsers provide no way to clear the cache or to explicitly check your site with an empty one (and Firefox actively pre-populates its cache, which includes this particular intermediate certificate). If you're a website developer or a system administrator and you check such a site, it's very likely that your regular browsers will have the necessary intermediate certificate cached. In Firefox, I think you have to test your website with a new scratch profile (and do it immediately after creating the profile before Firefox starts downloading intermediates on you).

Figuring out the actual problem took me rather a long time. First I spent a while trying some browsers in various environments, and then I thought that Windows on the machine might have an incomplete or damaged root certificate store. Only when I started peering very closely at Chrome's view of certificates on another Windows machine did I notice that the particular certificate was an intermediate certificate, not a CA certificate, and the penny dropped. I'm not sure a non-specialist could ever have diagnosed the problem, and even diagnosing the problem didn't make it easy to fix.

(The website's TLS certificate included an 'issuing certificate URL' (from RFC 5280 section 4.2.2.1), so a browser could have distinguished this from a completely unknown certificate authority if it wanted to.)

In retrospect, there was an important clue that I ignored and some things that I should have done. The important clue was that certigo, my usual tool for dumping certificate information, reported that it couldn't verify the certificate chain; at the time I wrote this off as 'maybe Fedora's root certificate list is odd', but I shouldn't have. The obvious thing I should have done was immediately run the site through the SSL Server Test, which would have immediately pointed out the problem. I could also have tried the site in a new scratch Firefox profile on my machine, which would probably also have pointed out the issue.

(I was blinded mostly because I was thinking of this as a client problem instead of a server one. Partly this is my Windows biases showing.)

In many environments, command line tools are useful for diagnosing this sort of thing because they (mostly) don't download or cache intermediate certificates. Instead, most of them verify the server's TLS certificate purely from the certificates provided in the TLS connection and the local CA trust store (which may include intermediate certificates but usually doesn't). If your browser works but something like certigo complains, generally either there's a missing intermediate or your system CA trust store is incomplete. I need to remember that for the next time around.

PS: At one point Firefox had an about:config preference that could be used to disable its cache of intermediate certificates, among other things, but that preference seems to have vanished in code rewrites since then.

from Hacker News https://ift.tt/U8aTcVg