Still, this type of oversight seems all too common even in large companies. I remember several cases from Fortune 500 companies in the past few years alone. What would be a good way to automate checking for them? Has anyone developed a tool designed specifically to avoid certificate expiry disasters?
LetsEncrypt renewal is supposed to be automated. [1]
I know of a company that hosted blogs for thousands of customers. They used LetsEncrypt, but the CTO considered automatic renewals a possible security risk, so they did it manually. Problem is, the expiration happened in a weekend and they "forgot" to update the certificates before that. Suffice to say that the next Monday wasn't pleasant. They automated after that.
I used manual renewal for LetsEncrypt for about 4 websites on other shared hosts & renewing them every 3 months was a pain; had to keep reminders and schedules just not to miss renewals until I synchronised their renewal schedules to batch (manual) renewing them.
I had automated renewal for 1 website on a cloud server, it was a one time effort, I never had to bother about SSL cert for that site and the most favourable of them all.
Their FAQ [1] recommends exactly that: renewing every 60 days.
[1] https://letsencrypt.org/docs/faq/#what-is-the-lifetime-for-l...
But now that you mention it, I wonder what's the opinion of security experts like tptacek on cert renewal automation.
To mitigate that you end-up building a series of privilege-restricted jobs flowing from the DMZ back into the internal network. And maintaining that might be more complicated than just manually renewing, depending upon the processes and architecture of the company.
I run Caddy (which uses acme-go/lego as its ACME provider) as a non-root user with no access to /etc at all. It seems to be running fine.
Of course that is also a kind of misconfiguration. The site has Debian security auto updates on but certbot is not among them. It should be forced to be updated. Furthermore there was no monitoring of errors in its log file.
Still it's not as simple as one believes Letsencrypt to be.
We issue certificates automatically if none is existing when connecting to a website and renew the certificates in batches 30 days before they expire. When renewing, we merge certificates/hostnames into bigger certificates with 90 hostnames so we don't have so many moving parts.
If renewal would break, however (as it did once or twice before), nothing bad would happen because on page load there would be a new certificate issued.
Another use case for the app I am developing! The basic idea: You can enter an item (i.e. "MyOwnShop Cert") into the list. From that time on, it will be tracked how much time passed since the item was entered or renewed (by clicking the renew button). The item with the longest time since entering/renewing is at the top of the list.
Compared to schedules and reminders it has the advantage that the item is not out of our mind once the reminder or schedule pasts. It just sits there dutifully and its timer keeps increasing.
I use it for keeping up with middle-term contacts ("Wow, I have not written Carl for 3 weeks?") and health-related issues. Logging in stuff that easily spoils would be another use case. And, apparently, cert renewals :)
acme-go/lego doesn't use HTTP validation unless you disable just about every other form of validation first. TLS-ALPN validation is much more likely, so port 443.
That said, it is very easy to allow software to bind to privileged ports without providing it root access; this has been solved for a very, very long time.
At the end of the day, someone needs to verify that new certificates gets acquired and installed before the old ones expire. Automation makes acquiring them less tedious, but not much for making sure someone pays attention.