Still, this type of oversight seems all too common even in large companies. I remember several cases from Fortune 500 companies in the past few years alone. What would be a good way to automate checking for them? Has anyone developed a tool designed specifically to avoid certificate expiry disasters?
I've never seen a bulletproof solution for organizational tasks that need to be done yearly.
If someone's in charge... and both they and their manager happen to leave in the same year... and whatever system they had in place to remember (probably their personal calendars) is gone... and the manager's manager has 1,000 other things to remember...
...how does an organization ensure the task still gets done?
I'm willing to bet Mozilla already does something like this but an engineer didn't set it up correctly for this certificate.
edit: not Europe, just UK and Japan apparently: https://www.zdnet.com/article/ericsson-expired-certificate-c...
Not sure that's viable for a signing certificate like this, but that's the way to solve it for the web PKI.
It took like two days before there was any kind of fix available, and they couldn't even roll it out automatically because the expiration had also disabled the auto-updating.
The problem is that Tree Style Tabs relies on userchrome.css edit to hide the tab bar, and when TST is forcibly removed there is no way to access the tabs, because that edited userchrome.css is still there. This is very disruptive. At least with the pre WebExtension addon TST itself hides the tab bar, so if TST is removed then the original tab bar comes back on automatically
https://github.com/eoger/tabcenter-redux/wiki/Custom-CSS-Twe...
#toolbar-menubar[inactive="true"] + #TabsToolbar {
visibility: collapse !important;
}With something almost stupidly simple and low-tech: checklists.
(I'm reading "The Checklist Manifesto" right now, and the points it makes seem to fit perfectly with everything you mention.)
LetsEncrypt renewal is supposed to be automated. [1]
I know of a company that hosted blogs for thousands of customers. They used LetsEncrypt, but the CTO considered automatic renewals a possible security risk, so they did it manually. Problem is, the expiration happened in a weekend and they "forgot" to update the certificates before that. Suffice to say that the next Monday wasn't pleasant. They automated after that.
https://pypi.org/project/check-tls-certs/
I run one daily from cron and have it email me a report with the days to expiration for the certs I’m responsible for, even for certs that auto renew. I don’t filter the email. Daily is not too frequent for it to go to my inbox, but frequent enough that I’ll notice if it doesn’t mail me. YMMV.
The problem comes if your keys ever get compromised or cracked all your historical traffic becomes vulnerable instead of just the most recent window.
Is anything more than a calendar reminder on the phone of someone important enough to shake the Earth and get it fixed For. Certain. needed? Like, say, the CEO, CTO, and CFO should at a minimum get a notification so they can ask if the refresh was done when necessary?
Also, Chrome is not immune to "crashes for everyone at the same time" bugs. Like that time when the start of daylight saving time made it crash for a full day (a quick search tells me it probably was https://bugs.chromium.org/p/chromium/issues/detail?id=287821).
I used manual renewal for LetsEncrypt for about 4 websites on other shared hosts & renewing them every 3 months was a pain; had to keep reminders and schedules just not to miss renewals until I synchronised their renewal schedules to batch (manual) renewing them.
I had automated renewal for 1 website on a cloud server, it was a one time effort, I never had to bother about SSL cert for that site and the most favourable of them all.
Their FAQ [1] recommends exactly that: renewing every 60 days.
[1] https://letsencrypt.org/docs/faq/#what-is-the-lifetime-for-l...
But now that you mention it, I wonder what's the opinion of security experts like tptacek on cert renewal automation.
What else would you expect for auto-updating software that relies on the internet to work? It's a monoculture attached to a firehose of disease.
This is exactly the same as "pushing out a security fix to all users," except it apparently wasn't intentional. You can't have one without the other.
You don't ship the signing keys with the certs, as that would be bad. ;)
Very few things check revocation, unfortunately - it puts an extra hop on the fast path of connecting to a server. OCSP stapling is pretty much the only thing a browser would care about - having the server fetch a signed OCSP response that is good for a limited period of time (say, hours), and send that along with the certificate during negotiation.
Or, you could just have the server fetch a certificate thats good for a limited period of time.
To mitigate that you end-up building a series of privilege-restricted jobs flowing from the DMZ back into the internal network. And maintaining that might be more complicated than just manually renewing, depending upon the processes and architecture of the company.
Certificate expiry really only exists to make money for CAs. It doesn’t solve any security problem that CRLs don’t already solve (and solve better). There’s lots of unsolved problems relating to ‘how do you make a reliable PKI’, but cert expiry is really just an unrelated business requirement for CAs.
Taking a look around, you’ll find lots of service providers, or tools you could use. But the main issue is all they do is tell a human being to do something, which they can still fail to do. Which is why automating cert rotation (with things like let’s encrypt or ACM) is arguably a better solution than monitoring it.
I run Caddy (which uses acme-go/lego as its ACME provider) as a non-root user with no access to /etc at all. It seems to be running fine.
Firefox still does normal OCSP requests, Chromes does not. So if you are a Chrome user, to my understanding, there is now way to know if the server certificate was revoked or not, other than OCSP stapling together with OCSP Must Staple. Additionally, both Chrome and Firefox ship a list of revoked certificates, but it may not be updated quickly enough and as far as i can tell it mostly contains roots and intermediates.
* Very short lifetimes get people to automate, preventing problems where one cert lasts long enough to lose the institutional knowledge around it.
* CRLs don't work. For performance you don't want to check for a revocation in serial with the request, and you don't want to block all browsing if the revocation list server is down. Revoking a cert will cover some users, but lots will still get "https://" and no warnings.
CRLs require maintenance and distribution of a list by a 3rd party. Creating an accurate, all-inclusive CRL of all website keys that your browser should reject is far, far from easy. (Case in point: "how many web sites are there?" Is not an easy question. )
Properly propagating such a list to any browser that might need it is another daunting task - less than 100% propagation means end users are exposed to security risks.
Certificate expiry is much more elegant: the client can check the certificate's validity himself, without relying on input from 3rd parties.
If certificates didn't expire, CRLs would (by now) be huge and growing enormously every day. They'd be so big that by the time you'd have downloaded one, it'd be outdated.
Of course that is also a kind of misconfiguration. The site has Debian security auto updates on but certbot is not among them. It should be forced to be updated. Furthermore there was no monitoring of errors in its log file.
Still it's not as simple as one believes Letsencrypt to be.
Not perfect, but I've added a TLS certificate extraction tool into a DPI that displays all visible certificates ordered by expiry date.
One could then mirror all one's site traffic to it and let it run in the background. Coupled with some alerting tool it would catch most of those cases I guess.
I could polish the tool a bit more if there is some interest, but anyone could do it as well.
See
https://github.com/rixed/junkie
and more specifically the plugin called 'sslogram'.
We issue certificates automatically if none is existing when connecting to a website and renew the certificates in batches 30 days before they expire. When renewing, we merge certificates/hostnames into bigger certificates with 90 hostnames so we don't have so many moving parts.
If renewal would break, however (as it did once or twice before), nothing bad would happen because on page load there would be a new certificate issued.
Another use case for the app I am developing! The basic idea: You can enter an item (i.e. "MyOwnShop Cert") into the list. From that time on, it will be tracked how much time passed since the item was entered or renewed (by clicking the renew button). The item with the longest time since entering/renewing is at the top of the list.
Compared to schedules and reminders it has the advantage that the item is not out of our mind once the reminder or schedule pasts. It just sits there dutifully and its timer keeps increasing.
I use it for keeping up with middle-term contacts ("Wow, I have not written Carl for 3 weeks?") and health-related issues. Logging in stuff that easily spoils would be another use case. And, apparently, cert renewals :)
But, this sharing carries a cost for user privacy, if I shard certs 16 ways then each CRL download gives me 4 bits of info about which sites you were visiting.
OCSP effectively takes this to the extreme, each lookup is tiny because it's just for one cert, but it gives away exactly which cert you cared about each time.
For any method, fail closed is user hostile and often a DOS vulnerability whilst fail open is another way for an attacker to use a revoked cert.
This is a big issue with on-line methods like OCSP as a MitM using a bad cert can probably block OCSP traffic as well.
CSLRs grow out of proportion, and leak information to the outside world.
Cert expiry serves as a backstop to these other revocation methods, and as a bonus ensures that simply forgetting about a cert cannot bite you 10 years later.
Failing closed means failure of a third party immediately breaks your site. Failing open means a MitM can simply block the CRL check.
OCSP stapling and the 'must staple' header are a lot better for privacy, and OCSP responses have some validity so at least a 5 hour outage of your CA doesn't bring your site down immediately.
It is still vulnerable to a DOS and trust on first use though.
My employer (Kynd.io) currently monitors public web sites for customers so we can flag e.g. "Hey this site cert expires in a week! If it's dead probably just switch it off, otherwise renew the certificate" and we're in the process of integrating CT but mostly so we can say "You already have a newer cert but need to go install it" in our How To Fix instructions.
Quixotically: make cert failure a randomised number, linearly related to how long ago the cert expired. This slowly introduces more and more failures, over a certain “grace period”, which makes the problem less of an extinction level event. It’s not a solution but it definitely would help.
acme-go/lego doesn't use HTTP validation unless you disable just about every other form of validation first. TLS-ALPN validation is much more likely, so port 443.
That said, it is very easy to allow software to bind to privileged ports without providing it root access; this has been solved for a very, very long time.
Certificates that aren't from the Web PKI almost invariably won't be logged. Most logs explicitly refuse everything except certs from the Web PKI so as not to be burdened storing garbage. So this won't find certs issued by the custom OpenSSL CA on that one guys Linux laptop.
Not all Web PKI certs are logged. There is no BR obligation and no root store programme rule that requires logging. The only things in place that strongly encourage logging are the Chrome and Safari policies. For systems that aren't designed to be accessed with a web browser or, much more rarely, enterprises that have persuaded themselves only IE is authorised anyway, the certs might deliberately not be logged. Yes there are (small) CAs doing this in the Web PKI, on purpose, in 2019.
Once you do this, the only checklist that matters are procedural checklists to add a new client or new cert to the renewal notification list. When you use a standard group email for all cert purchases, that one becomes tough to miss.
In my 7 years of being involved, we never missed a cert renewal with this process for ~300 client sites with multiple or wildcard certs.
Short lived certs are quite obviously better from a security perspective, but the security difference between a certificate that expires in five years, and one that expires never is irrelevant.
All it being new means is that depending on your risk ratio you need to decide whether updates to the software need testing or whether you need to invest in your own solution - or, how about just wait until it matures and keep the old process until then.
Waiting doesn't invalidate the premise either. It just means you lack the resources to implement it safely and that's ok.
The information leakage of CRLs is stating to the public that a cert needed to be revoked.
Obviously, a compromised cert that will expire in 5 years is horrible. However, a non compromised cert you are no longer using that will never expire is more off a risk than a disused cert that will expire in a year. Not to say you should leave the one year cert lying around. However, there is no desire to put the one year cert on a pre-shipped CLR.
Register a domain, get a certificate lasting forever, let the domain expire and somebody buy it. Then somehow redirect all or part of the traffic to that domain to your own server with a valid certificate. Chances are that few people will notice something has changed in the details of the certificate.
However you'll have left traces all over the place: credit cards, phone numbers, etc.
At the end of the day, someone needs to verify that new certificates gets acquired and installed before the old ones expire. Automation makes acquiring them less tedious, but not much for making sure someone pays attention.
Apache and nginx both shipped OCSP stapling implementations that are very bad, awful enough that for almost anyone I'd say "No, don't enable that" rather than try to explain how they need to use it and get them to a place where it's useful and safe. Adam Langley wrote years ago about how to do this correctly, and there does seem to be a little bit of movement in the correct direction at Apache, but the situation remains pretty poor.
Corporations are often not very good at putting the right people in these roles but good ones are invaluable. Since the Marvel Universe is everywhere, Pepper Potts is the archetype in that setting to give you an idea of why you'd need people like this. Tony Stark would be "too busy" to renew the certificates, but Pepper would make sure it gets done.