←back to thread

All extensions disabled due to expiration of intermediate signing cert

(bugzilla.mozilla.org)
1318 points xvector | 10 comments | 04 May 19 01:31 UTC | HN request time: 0s | source | bottom
Show context
needle0 ◴[04 May 19 01:56 UTC] No.19823806[source]▶
I’ll still keep using Firefox since I recognize the importance of browser diversity and the hazards of a Chrome monoculture (that and vertical tabs), but, yikes.

Still, this type of oversight seems all too common even in large companies. I remember several cases from Fortune 500 companies in the past few years alone. What would be a good way to automate checking for them? Has anyone developed a tool designed specifically to avoid certificate expiry disasters?

replies(18): >>19823825 #>>19823829 #>>19823831 #>>19823840 #>>19823848 #>>19823861 #>>19823913 #>>19823994 #>>19824009 #>>19824223 #>>19824243 #>>19824298 #>>19824668 #>>19824724 #>>19824795 #>>19824840 #>>19824927 #>>19825103 #
revvx ◴[04 May 19 02:36 UTC] No.19823994[source]▶
> Still, this type of oversight seems all too common even in large companies. (...) Has anyone developed a tool designed specifically to avoid certificate expiry disasters?

LetsEncrypt renewal is supposed to be automated. [1]

I know of a company that hosted blogs for thousands of customers. They used LetsEncrypt, but the CTO considered automatic renewals a possible security risk, so they did it manually. Problem is, the expiration happened in a weekend and they "forgot" to update the certificates before that. Suffice to say that the next Monday wasn't pleasant. They automated after that.

[1] https://letsencrypt.org/about/

replies(9): >>19824056 #>>19824264 #>>19824303 #>>19824403 #>>19824729 #>>19824926 #>>19825434 #>>19825826 #>>19826191 #
mc32 ◴[04 May 19 02:50 UTC] No.19824056[source]▶
So did they conclude it wasn’t a security concern or did they conclude the security risk was worth the uptime?
replies(2): >>19824094 #>>19824258 #
revvx ◴[04 May 19 03:32 UTC] No.19824258[source]▶
When pressed, they admitted it was just "gut feeling". The team audited a couple ACME clients and couldn't find anything to justify not automating.
replies(1): >>19824591 #
1. dingaling ◴[04 May 19 05:12 UTC] No.19824591[source]▶
Having a root process with write-privileges to /etc on production machines and also able to communicate over the Internet definitely is a security risk.

To mitigate that you end-up building a series of privilege-restricted jobs flowing from the DMZ back into the internal network. And maintaining that might be more complicated than just manually renewing, depending upon the processes and architecture of the company.

replies(1): >>19824753 #
2. Whitestrake ◴[04 May 19 06:09 UTC] No.19824753[source]▶
Why would a process need to run as root or have write privileges to /etc in order to automate LetsEncrypt renewals?

I run Caddy (which uses acme-go/lego as its ACME provider) as a non-root user with no access to /etc at all. It seems to be running fine.

replies(2): >>19824866 #>>19825032 #
3. tedunangst ◴[04 May 19 06:48 UTC] No.19824866[source]▶
Depends on setup, but frequently private keys are inaccessible to the web server worker process. (Which starts as root, loads keys, drops privs, etc.)
replies(1): >>19825026 #
4. tialaramex ◴[04 May 19 07:36 UTC] No.19825026{3}[source]▶
Most popular ACME (Let's Encrypt) clients allow you to provide a CSR instead of generating the keys themselves. That means a bunch more work for you, but if you're worried about this, that's what you should do. Have your safe (even manual if you insist) process make keys, make CSRs for the keys, and put those somewhere readable. The ACME client will hand them over to the CA saying "I want certs corresponding to these CSRs" without needing access to your TLS private keys at all.
replies(1): >>19825034 #
5. rocqua ◴[04 May 19 07:38 UTC] No.19825032[source]▶
Using http renewal requires listening on port 80 which, by default, requires root.
replies(3): >>19825104 #>>19825110 #>>19826112 #
6. rocqua ◴[04 May 19 07:39 UTC] No.19825034{4}[source]▶
That does mean you aren't automatically rotating keys anymore.
replies(1): >>19826292 #
7. Whitestrake ◴[04 May 19 08:03 UTC] No.19825104{3}[source]▶
This is technically true, but contextually lacking.

acme-go/lego doesn't use HTTP validation unless you disable just about every other form of validation first. TLS-ALPN validation is much more likely, so port 443.

That said, it is very easy to allow software to bind to privileged ports without providing it root access; this has been solved for a very, very long time.

8. ◴[04 May 19 08:04 UTC] No.19825110{3}[source]▶
9. revvx ◴[04 May 19 12:53 UTC] No.19826112{3}[source]▶
You can just use the web server that is already running on the machine.

You (normally) don't want downtime in your website, so you just let your regular webserver serve the acme challenge instead of stopping it.

10. revvx ◴[04 May 19 13:25 UTC] No.19826292{5}[source]▶
If you trust your automation, you put private key rotation into it.

If you don't trust it your automation, you rotate the keys manually, as you would normally.

There are no valid reasons to throw the baby away with the bathwater.