All extensions disabled due to expiration of intermediate signing cert

I think this fail-closed behavior is more of a security issue than the one it is trying to solve. All of my security add-ons - Privacy Badger, NoScript, Decentraleyes, and many more were disabled. Even worse, it happened without notice to the user.

One moment I was browsing the internet (just barely) secured by these add-ons, and the next moment, all of them disappeared (without warning) and I only noticed when I saw my password manager was missing.

I like FF, don't get me wrong, but this is going to absolutely fucking destroy user trust in Mozilla. This kind of incompetence, on a browser scale, is breathtaking.

I'm not sure this cert is used with the PW manager?

I think HN penalizes non-link posts (or people are flagging it because they think it's just someone asking for tech support).

Firefox password storage isn't even encrypted by default, last I checked.

Still, this type of oversight seems all too common even in large companies. I remember several cases from Fortune 500 companies in the past few years alone. What would be a good way to automate checking for them? Has anyone developed a tool designed specifically to avoid certificate expiry disasters?

I've never seen a bulletproof solution for organizational tasks that need to be done yearly.

If someone's in charge... and both they and their manager happen to leave in the same year... and whatever system they had in place to remember (probably their personal calendars) is gone... and the manager's manager has 1,000 other things to remember...

...how does an organization ensure the task still gets done?

Edit: wow, downvotes? Care to explain what I'm missing?

I'm willing to bet Mozilla already does something like this but an engineer didn't set it up correctly for this certificate.

edit: not Europe, just UK and Japan apparently: https://www.zdnet.com/article/ericsson-expired-certificate-c...

They'd better have the best post mortum ever, possibly with someone being fired.

Arguably these two goals are incompatible. :)

Not sure that's viable for a signing certificate like this, but that's the way to solve it for the web PKI.

Temporary work around till the cert gets fixed: set "xpinstall.signatures.required" to false

Mistakes happen, it's okay. But users should be empowered to work around them.

Warning about something that can't be verified is one thing, but automatically shutting things off -- particularly things that could affect security and privacy -- is a Windows 10 level of unacceptable interference.

It took like two days before there was any kind of fix available, and they couldn't even roll it out automatically because the expiration had also disabled the auto-updating.

The problem is that Tree Style Tabs relies on userchrome.css edit to hide the tab bar, and when TST is forcibly removed there is no way to access the tabs, because that edited userchrome.css is still there. This is very disruptive. At least with the pre WebExtension addon TST itself hides the tab bar, so if TST is removed then the original tab bar comes back on automatically

Before the forced code signing, before the automatic updates, Mozilla or any other organisation's mistakes would not have such dramatic effects; now, they have the power to basically break almost all their userbase nearly instantly, and that is what worries me the most.

The issue is that if you leave any sort of lever that reduces security, it will be abused by bad actors. This is why browsers are having ever decreasing ways to bypass security and have full access. It is annoying, but at the end of the day, protecting 99.999% of the users trumps what us power users want.

> Temporary work around till the cert gets fixed: set "xpinstall.signatures.required" to false

https://news.ycombinator.com/item?id=19823879

However, if you're running the Stable or Beta version, it will only work under Linux. On Windows and MacOS you'll need to download Nightly or the Developer Edition.

To fix this on MacOS I did the following:

1. Downloaded and installed Firefox Nightly

2. Ran /Applications/Firefox\ Nightly.app/Contents/MacOS/firefox-bin --profilemanager

3. Changed the profile to "default" so my normal Firefox profile would be used

4. Started up Firefox Nightly, opened about:config, then set xpinstall.signatures.required to false

Not sure if it's a good idea to use my default profile in Nightly. It might be a wiser idea to copy it instead.

Or is HTTPS / LetsEncrypt too big to fail? HTTPS still always a good choice? I see…

I think we'll all live. No need for the chicken little act.

https://github.com/eoger/tabcenter-redux/wiki/Custom-CSS-Twe...

  #toolbar-menubar[inactive="true"] + #TabsToolbar {
    visibility: collapse !important;
  }

It is horribly paternalistic to advocate for keeping users ignorant, unlearning, and --- dare I say it --- easily manipulated.

I will refrain from mentioning again that infamous Franklin quote. I am frankly very fucking pissed off by this authoritarian walled-garden trend, and vehemently oppose anyone who helps this industry put the nooses around the necks of others as well as their own.

For a piece of open source software you really have very little control with firefox. It really sucks that the alternatives are worse.

This, likely for almost all of their users, creates more of a security problem than signature checking actually solves. For me noscript no longer works which is (IMO) a critically important extension (between mozilla taking away the disable javascript button and spector.)

With something almost stupidly simple and low-tech: checklists.

(I'm reading "The Checklist Manifesto" right now, and the points it makes seem to fit perfectly with everything you mention.)

The fundamental problem here is the system (code signing.) It's a political thing with security being the excuse. They want control of a platform for business reasons.

I still don’t want to have to understand everything I ever touch, even if I could.

No one needs to be fired for a single instance of a particular mistake. If this happened multiple times, then I would be on board with firing someone.

LetsEncrypt renewal is supposed to be automated. [1]

I know of a company that hosted blogs for thousands of customers. They used LetsEncrypt, but the CTO considered automatic renewals a possible security risk, so they did it manually. Problem is, the expiration happened in a weekend and they "forgot" to update the certificates before that. Suffice to say that the next Monday wasn't pleasant. They automated after that.

[1] https://letsencrypt.org/about/

I do think that in the future, it will be imperative for everyone to have some level of technological literacy above what is currently the average. And I'd like to work to get to that point, instead of taking all the tools away because they're too dangerous.

Also, sensible defaults are good! Hiding dangerous settings is also good! What's not okay is making those settings completely unavailable. At least in Firefox's case you have the option to recompile the source code, but that should not be the only recourse...

This is just plain bad.

Or you fire the scapegoat because of a broken system that allowed one person to make a mistake?

https://pypi.org/project/check-tls-certs/

I run one daily from cron and have it email me a report with the days to expiration for the certs I’m responsible for, even for certs that auto renew. I don’t filter the email. Daily is not too frequent for it to go to my inbox, but frequent enough that I’ll notice if it doesn’t mail me. YMMV.

Saved me tons of ultimately pointless thrashing.

I thought that the way signing works in general is that the signer issues a certificate for the thing being signed (domain, code, whatever) that contains identifying information for the thing signed (host name for an SSL certificate, checksum of the code for a code signing certificate), the valid from and valid to dates of that certificate, and assorted other information, and either a reference to or a copy of the signer's certificate, and it signs the whole issued certificate with the signer's certificate.

Someone checking the signed thing is supposed to consider it validly signed if:

1. The date is in the valid range for the signed thing's certificate,

2. A check of the signature of that certificate against the signing certificate passes,

3. The signing certificate is recognized as being from an issuer considered trusted by the checker,

4. Neither the signed thing's certificate nor the signing certificate have been revoked, and

5. The signing took place during the valid date range of the signing certificate.

Note there is no "the date of the check is in the valid date range of the signing certificate". A signing certificate expiring should not invalidate things signed by it. It should just prevent signing anything else with it.

So why is a signing certificate expiring for Firefox breaking already signed extensions? Shouldn't it just be stopping new versions of extensions from being signed?

If we're going to be authoritarian I would rather ban anyone who doesn't understand that from connecting to the internet then have a broken walled garden.

My concern is around non-technical users (the group, mind you, that Firefox has been spending marketing dosh on courting recently with Quantum and all) who don't have as compelling reasons for not just switching back to Chrome. In the last hour, I've gotten several phone calls from family members asking me why the browser I convinced them to use is broken. I don't have a good answer, because platitudes about surveillance and muh freedoms don't count for shit when your grandma just wants to get rid of the ads on the local newspaper site.

I'm personally going nowhere and deeply appreciate Mozilla for all the work on FF and friends, occasional fuckups aside, but I don't think this is going to be a non-event for a browser that's been desperately fighting to regain market/mind-share.

I'm referring to traditional code signing, which I assume Firefox extensions are more similar to --- the goal being to ensure that some data has not changed since it was signed, and only the validity of the certificate at the time the data was signed is meaningful; even after the certificate expires, a signature created when it was valid still asserts that the data it signed has not changed.

When I opened my alt profile, extensions still worked, but I can't install updates—including for HTTPS Everywhere.

What's up with that?

    $ date
    Fri May  3 22:45:22 EDT 2019

    $ date --utc
    Sat May  4 02:41:47 UTC 2019

    $ firefox --version # Installed from arch repositories
    Mozilla Firefox 66.0.3

    xpi.signatures.required true
    app.update.lastUpdateTime.xpi-signature-verification 1556920447
    extension.update.enabled true

Edit: I think I know why. It checks the signatures daily, and the timing works out so it hasn't checked since the cert expired for me. Just luck, it will break within the next 21 hours for everyone. From the source code:

    const XPI_SIGNATURE_CHECK_PERIOD      = 24 * 60 * 60;

    [...]

    timerManager.registerTimer("xpi-signature-verification", () => {
      XPIDatabase.verifySignatures();
    }, XPI_SIGNATURE_CHECK_PERIOD);

The model you're suggesting is closer to the "timestamping" one commonly used in code signing (IIRC Windows and Mac both do this) where a third party that's particularly trusted to handle key material well long-term gives you a second signature over the message "I saw this signature at this time" (effectively they are analogous to a notary or witness for real-world signatures). Then you can trust that signatures from expired signing certs were actually made in the past, and not by an attacker who got hold of the key. That is, without timestamping you have no proof of #5 in your list.

(I suppose you could do this now for the SSL PKI with Certificate Transparency logs.... it isn't exactly what they were built for but it's probably sound.)

I basically can't (OK, won't) browse anything except HN until they do.

Why do my personal extensions need to be hooked into some third party service that can go out at anytime?

What I'd like to see is a post-mortem, followed by an explanation of how they'll prevent the mistake from being made again in future.

If we're going to assume that software is right and the user is wrong 100% of the time, then the software needs to actually be right 100% of the time. Unfortunately, our software isn't that robust, and it never will be.

Does it only occur after a restart or?

This has probably happened to every major cloud provider and countless companies at least once. Certs are hard.

Should Mozilla have had monitoring on their cert expiration? Yes. Will they after this? Probably. Is any one person ever at fault for something like this? No.

Firefox is an open source project. You're welcome to contribute and make things better.

That is absolutely complicated for the vast majority of the world's internet users. No one else is my family would understand what the hell "privileged code" means and shouldn't have to.

Well no because they won't accept a patch that lets us plebs turn off the signed extension requirement.

Something we could refer to when discussing possible pitfalls?

That might seem rather extreme, but the fact that this situation was even possible was a consequence of a series of bad decisions over an extended period of time about the required behaviour of new versions of Firefox, combined with technical failures that betray fundamental weaknesses in the whole system design. Whoever was ultimately responsible for those failings demonstrably isn't competent to run something of this importance and should probably either implement immediate and dramatic changes to the relevant policies and technical details or consider their position. Anything less is surely going to damage trust, which is something Firefox can ill afford when it's already in danger of being reduced to a niche product rather than a mainstream browser.

But we're outsiders looking in and don't know what's going on at this point. That's why I used the qualifier "possibly." It's quite possibly it wasn't incompetence.

If you don't understand it, don't touch it. The default settings should work for most users. There can even be a warning against touching without understanding, like with Firefox's about:config. The offensive thing is preventing users from touching even if they do understand.

> diox commented 2 hours ago

> I'm locking this like I did in #851 because no new information is being added. We're aware and we're working on it.

I mean, two hours? WTF.

Honestly that's one of the most successful things you can expect out of a failure of this magnitude.

Still, it's no worse a calamity than most everyone else presents to the world from time to time. I shall stick to my Firefox, partly influenced by the complete absense of any viable alternative out there.

By the way: Writing this a few hours after midnight, May the fourth, local time. Every single one of my 33 Firefox addons is active and working just fine. Wating to see. Breaking out my Waterfox, just in case.

[Edit: May fifth -> fourth]

Also, wow, the web has a ton of ads. I've been running uBlock origin so long I forgot how bad it had gotten :(

That said, existing signatures that have already been verified, for existing extensions, should still be trusted.

Asking people to either give up control of their software (ie, walled garden release versions) or use buggy and insecure software Dev/Nightly/etc is not acceptable.

It's why I switched to a freedom respecting Firefox fork as soon as they announced walled garden extension signing in Firefox 37.

The problem comes if your keys ever get compromised or cracked all your historical traffic becomes vulnerable instead of just the most recent window.

Is anything more than a calendar reminder on the phone of someone important enough to shake the Earth and get it fixed For. Certain. needed? Like, say, the CEO, CTO, and CFO should at a minimum get a notification so they can ask if the refresh was done when necessary?

:/

  jtl@laptop-linux:~$ TZ=UTC date --date="@1556919381"
  Fri May  3 21:36:21 UTC 2019

Discipline? Experience? PIP?

Also, Chrome is not immune to "crashes for everyone at the same time" bugs. Like that time when the start of daylight saving time made it crash for a full day (a quick search tells me it probably was https://bugs.chromium.org/p/chromium/issues/detail?id=287821).

We check our warning system (set up to detect suspicious logins, incidentally also catches any users who've been locked out because they forgot their password), and his last login attempt took a total of two tries.

If it hasn’t broken yet for you, I think (but I’m not very much not sure) setting that preference to 1556940100 should keep it working until 24 hours from now. And if you keep updating that value every 23 hours to the output of date '+%s' until it is fixed via a firefox update it should keep working forever.

I think you need to restart the browser as well after updating the preference for the above idea to work.

Without timestamping the expired cert always would have caused problems, even if it was replaced early and correctly: Every add-on would still need to be signed again with the new replacement certificate and shipped to all users. It's not as easy as just replacing the certificate on some server.

Well, this is still what has to happen: replace the certificate, ship that new certificate[1], re-sign every add-on, ship every add-on to every user.

Now, in order to ship new versions of the add-ons, you probably will have to bump the add-on version numbers as well. Which can have further unintended consequences.

[1] Incorrect, see blow; it is my understanding that the certificate in question is baked into the browser itself, with no way to push updates just for the certificate remotely other than shipping an entire new Firefox build. Well 6 new builds: esr, stable, dev, beta, nightly, unbranded. Gonna be a fun night for a lot of mozilla folks... Well, a night is not gonna be enough...

I might be wrong tho, and misunderstood something.

EDIT I was wrong (https://news.ycombinator.com/item?id=19824520), the expired cert is not baked into the browser, just into the add-on package files. No need for new Firefox binaries, after all. Still, they have to resign all add-ons and ship new versions.

Adjust the qualifier at the end depending on your platform. On Windows, it might be apps that present a UAC dialogue—or maybe just remove the qualifier, since Windows doesn't do much sandboxing by default.

I used manual renewal for LetsEncrypt for about 4 websites on other shared hosts & renewing them every 3 months was a pain; had to keep reminders and schedules just not to miss renewals until I synchronised their renewal schedules to batch (manual) renewing them.

I had automated renewal for 1 website on a cloud server, it was a one time effort, I never had to bother about SSL cert for that site and the most favourable of them all.

If there's a privilege level that allows for one but not the other, that sounds like something Mozilla should fix.

https://bugzilla.mozilla.org/show_bug.cgi?id=1528738

It's stuff like this that makes me unhappy with mozzilla. User's who know what they are doing should be permitted to do so. Warn them here be dragons or whatever, but it's ultimately their choice.

So here is one example: https://stackoverflow.com/questions/18248020/certificate-has...

You say walled garden, I see what random WebExtensions people install on their work laptops and think "yeah maybe someone policing this thing isn't the worst thing". But most importantly: it sounds like it's not actually a problem for you?

Preferences > Privacy and Security > Strict

Everyone's extensions broke. Including security ones. Including the ones bundled into the TOR browser. And end-users can't fix it. Because Mozilla decided that it was too dangerous to let users choose what extensions to run for themselves. This is an excellent moment to be upset.

Their FAQ [1] recommends exactly that: renewing every 60 days.

[1] https://letsencrypt.org/docs/faq/#what-is-the-lifetime-for-l...

[1] https://caddyserver.com

[2] https://httpd.apache.org/docs/2.4/mod/mod_md.html

Example: https://subdavis.com/Tusk/

But now that you mention it, I wonder what's the opinion of security experts like tptacek on cert renewal automation.

If you want your freedom from reviewed extensions: fine, get an unbranded Firefox, or Developer edition, and you get that.

https://i.imgur.com/hUe7Nyn.png

Very much this. People are often too quick to forget who their customers are and what they really want.

In this case, dropping the extra control/ignoring power users is probably saving a lot of non-power users from shooting themselves in the foot in the vast majority of cases. Pilots (should be) 100% power users. The average operator of a browser is somewhere on the opposite end of the spectrum.

Any real system will have things go horribly wrong for some subset of users on a regular basis. It's impossible to be all things for all people for all situations, so you have to choose your battles.

And if they were good at planning, this would not have happened in the first place. At least give us a button or something for "I don't give a shit if it isn't signed, enable it"

that's kinda the problem. there's plenty of reasons to be "with" firefox still, but you shouldn't need reasons other than it's the best browser. when it starts requiring loyalty to be a user, that's a big problem.

What else would you expect for auto-updating software that relies on the internet to work? It's a monoculture attached to a firehose of disease.

This is exactly the same as "pushing out a security fix to all users," except it apparently wasn't intentional. You can't have one without the other.

The goal of recurring task is to get people's attention to review and perform tasks the happen periodically. It could be as simple as reviewing it and marking it done. They will show up as part of the bug report to the assignees, so they can at one place see all the bugs, feature requests, tasks, and recurring tasks.

Cert renewing would fall under the recurring task category.

If you have a well-designed system that only works with encryption, then sure, but this idea of using the same mistaken systems as the WWW clearly doesn't work well.

I've never seen a Tor Hidden Service fail because of something expiring.

Much of this nonsense about encrypting everything, without reason and excuse, is to protect advertisements from being modified.

That this hit Tor Browser and disabled NoScript is damning, but I already disable JavaScript in about:config and I'm not even using a version of Firefox this new, anyway.

I can't tell if my opinion of Mozilla is lower or if it can't get lower.

Try turning it off. I got rid of ublock after arstechnica complained about a lot of their users blocking ads years ago and it honestly isn't that bad. Every once in a while I do back out of a page for maxing out one of my cpu cores but otherwise, nothing ever bad happens. With ads: either it takes me half a second to tell I'm not interested in an ad, or I actually am interested and i follow the ad because I am interested and I want to support the website.

The alternative is websites charging insane amounts of money with paywalls (Wall street journal has their "best" price for 12 months at $360 a year). That is horrible because it means only rich people can pay for high quality news as ads are one of the most progressive forms of payment (rich people ads are way more valuable than poor peoples and yet everyone gets the same quality services/news with the ad model despite their income/net worth).

Open the browser console by hitting ctrl-shift-j

Copy and paste the following code, hit enter. Until mozilla fixes the problem you will need to redo this once every 24 hours:

    // Re-enable *all* extensions

    async function set_addons_as_signed() {
        Components.utils.import("resource://gre/modules/addons/XPIDatabase.jsm");
        Components.utils.import("resource://gre/modules/AddonManager.jsm");
        let addons = await XPIDatabase.getAddonList(a => true);

        for (let addon of addons) {
            // The add-on might have vanished, we'll catch that on the next startup
            if (!addon._sourceBundle.exists())
                continue;

            if( addon.signedState != AddonManager.SIGNEDSTATE_UNKNOWN )
                continue;

            addon.signedState = AddonManager.SIGNEDSTATE_NOT_REQUIRED;
            AddonManagerPrivate.callAddonListeners("onPropertyChanged",
                                                    addon.wrapper,
                                                    ["signedState"]);

            await XPIDatabase.updateAddonDisabledState(addon);

        }
        XPIDatabase.saveChanges();
    }

    set_addons_as_signed();

https://community.letsencrypt.org/t/pip-error-with-certbot-a...

I wonder if this bit Mozilla and caused this issue.

https://github.com/icing/mod_md/wiki/Migration

If ads weren't doubling as tracking beacons and the occasional malicious drive by download, that certainly would be an option.

https://catless.ncl.ac.uk/Risks/

Thanks :)

You don't ship the signing keys with the certs, as that would be bad. ;)

[0] https://ctadvisor.lolware.net/

Very few things check revocation, unfortunately - it puts an extra hop on the fast path of connecting to a server. OCSP stapling is pretty much the only thing a browser would care about - having the server fetch a signed OCSP response that is good for a limited period of time (say, hours), and send that along with the certificate during negotiation.

Or, you could just have the server fetch a certificate thats good for a limited period of time.

Sadly, I have to agree that this feels like a big blow to user trust.

User trust is not really just about respect or values; it definitely also includes things like performance and reliability. The average user, right now feeling powerless, might even feel anger towards Mozilla for this - after all, they already downloaded the extension, why would they all just stop working behind their backs? They don't understand what CAs are or why certificates expire. People don't frankly care what place your heart is in when they are angry about something. Perhaps people are being dramatic, but that's normal. People are pretty darn dramatic about Chrome, too.

Meanwhile... I use Firefox everywhere, and I've lost my password manager, adblocking, security-related extensions, etc. all in one go, and the only solutions I'm aware of involve disabling extension signing. Gotta admit, even though I will probably continue using Firefox after this, that it certainly is a bummer.

You just described something bad.

That's true, sort of. How often do you let people make huge mistakes before you decide that maybe they are just not apt for the position that they've been promoted to and Peter was right? Once? Twice? And unlimited amount, as long as it's never the exact same mistake?

Then the issue becomes: how to get the new certificate to a few hundreds million users?

If it was a certificate on some server, just replace it there, done. Client software will just pick it up. But not here. A copy of the certificate is shipped in every add-on package file. Oops. Now you have to re-sign all add-ons with the new certificate. And get those resigned files to the users.

Essentially it works like this (which is a slightly modified jar/apk signing mechanism):

- An add-on package is a zip file and other than the actual files there is also a list of known-good hashes of those files in a file called "manifest.mf" in the META-INF folder

- Then there is a file "META-INF/mozilla.sf" giving hashes of "manifest.sf"

- And finally, there is "META-INF/mozilla.rsa", which is a DER-encoded pkcs7 signature and two certificates. The signature verifies "mozilla.sf" was not tampered with and still is the same as when it was signed by mozilla. Which in turn verifies the known-good hashes are still proper.

- The signature is made with a generated certificate, the first one included in "mozilla.rsa". E.g. "CN=uBlock0@raymondhill.net" in case of uBlock.

- The "CN=uBlock0@raymondhill.net" certificate was issued by an intermediate certificate "CN=signingca1.addons.mozilla.org". This "CN=signingca1.addons.mozilla.org" is the second certificate in mozilla.rsa. It says "Validity Not After : May 4 00:09:46 2019 GMT". Oops. This is where the chain breaks now!

- "CN:signingca1.addons.mozilla.org" was issued by "CN=root-ca-production-amo". This root certificate is baked straight into the browser and not part of mozilla.rsa.

Therefore, it is not enough to issue another intermediate certificate (e.g. "CN=signingca-number-two.addons.mozilla.org"), but you have to actually generate a new "CN=uBlock0@raymondhill.net" (or whatever) signed by this new certificate, put those two certificates and a new signature of "mozilla.sf" based on those new certificates into a new mozilla.rsa FOR EACH add-on and ship updated add-on files.

PS:

Try it yourself... Extract some addon package (it's a zip file). Then:

    openssl pkcs7 -in META-INF/mozilla.rsa -inform DER -print

While I agree with you two assuming the bridge is over water and not too high, there are real consequences that cannot be reversed. I cannot unsee the ads I saw in the past few minutes before switching to nightly.

And yet every other major browser vendor has punched their users with far worse catastrophes of privacy, security, ripping away features, breaking features, and general shitheaddedness.

Switching browsers because of this incident is like ordering a burger at your favourite restaurant and one time it comes out without the meat patty, so in protest you switch to a crappy alternative restaurant that has had a long history of health code violations.

I'm glad you have software/vendors you feel you can trust. I definitely don't feel that way about most software anymore. I do think you are being a bit hyperbolic regarding other browser vendors, but to each their own, I don't know what trying to argue about that would solve for anyone.

I like it! Will talk about this with my team lead!

Funding the Internet? What you're talking about (ads) is a revenue stream for what amounts to a handful of websites. google.com, amazon.com, ycombinator.com, reddit.com, thefacebook.com, tweeter.com, etc. could all go offline right now and the Internet would still be here.

To mitigate that you end-up building a series of privilege-restricted jobs flowing from the DMZ back into the internal network. And maintaining that might be more complicated than just manually renewing, depending upon the processes and architecture of the company.

diox commented 4 hours ago

I'm locking this like I did in #851 because no new information is being added. We're aware and we're working on it. This conversation has been locked as spam and limited to collaborators.[1]

Bug 1548973 (armagadd-on-2.0) All extensions disabled due to expiration of intermediate signing cert NEW Unassigned (Needinfo from 3 people)

Kevin Brosnan [:kbrosnan]

We have confirmed this issue. Extra comments about this being broken will not advance this bug to being fixed.[2]

Mozilla just left their entire user base unprotected against ads, trackers, and some hostile code. Then they insult their users.

Undoing the damage is hard. First, they have to update their signing certificate. Then they have to re-sign all the add-ons. Then users have to reload all the addons. Then, something users won't do - remove all the tracking cookies, etc. that slipped in while Firefox was broken.

[1] https://github.com/mozilla/addons/issues/978

[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1548973

That way even if the business messed up, they would have a heads up from users to fix it before d-day when everything stops working. This includes website certs and addon signing certs and any intermediaries.

This could have been prevented by someone putting the expiration date on the team shared calendar with a 60 day alert.

https://traefik.io

I'm pretty sure Mozilla will implement a fix in a way that users only have to update their browser, not do anything to all their addons.

They are working on it, and seeing 1000s of “me too” comments in the issue isn’t going to make things better for anyone. Least of all their customers, who, when they do update the issue with more info, won’t have to wade through pages and pages of noise before they get to the actual update from Mozilla.

"We accidentally uploaded all your HTTP requests to our servers, but we will definitely fix that in next addon version!~"

[1]: https://arstechnica.com/?post_type=post&p=1340459

The npm self-signed certificate fiasco of early 2014 springs immediately to mind.

Otherwise I'm not bothered. I won't be switching as long as this gets resolved within the next few days.

This should not be possible.

Worse, had they not taken the paternalistic, nanny-like stance that you can't even disable the signing checks, I could roll out a script that would make this a non-issue for my users. But no, thanks Mozilla for ruining my Monday.

Might not be the most substantive comment I could possibly make in the circumstances, but I'm pissed. The only appropriate response feels like a string of infuriated profanity directed at their incompetence and decision-making.

Note: I am told that Developer channel uses a separate profile, but there are instructions below showing people how to override that, at which point this warning becomes relevant once again.

With this reduced threat model, it's easy to simply keep existing pre-installed extensions available, and disable updates. Your only problem is if a pre-installed extension is malicious or has a vulnerability, it will remain.

There is a possibility that Mozilla implemented their backwards code-signing model on purpose — for example, it allows them to oust unwanted extensions without explicitly recalling their certificates. But personally I think that they just didn't give the matter enough thought.

Certificate expiry really only exists to make money for CAs. It doesn’t solve any security problem that CRLs don’t already solve (and solve better). There’s lots of unsolved problems relating to ‘how do you make a reliable PKI’, but cert expiry is really just an unrelated business requirement for CAs.

Taking a look around, you’ll find lots of service providers, or tools you could use. But the main issue is all they do is tell a human being to do something, which they can still fail to do. Which is why automating cert rotation (with things like let’s encrypt or ACM) is arguably a better solution than monitoring it.

Probably the correct behavior is to have some sort of semi-annoying popup when it expires, and then only a week later do the full blocking. You need to strike the right balance of making it annoying enough that it can't be ignored by everyone (otherwise you just have the exact same problem, just delayed a week) and that fear of it happening is a sufficient motivator to stop people lazily relying on the grace period, but also not too annoying that it makes a lot of people quit. You also want to avoid permission fatigue.

Fixed for now by switching to Firefox Nightly and disabling signing.

I run Caddy (which uses acme-go/lego as its ACME provider) as a non-root user with no access to /etc at all. It seems to be running fine.

I was able to piece together most of my compact dark interface theme [1] with userChrome.css by sacrificing the all-tabs menu for its JS binding, but the all-tabs helper addon is a shadow of what it once was, and the Private Tabs addon is dead with no hope of revival due to the lack of a WebExtension API [2]. I can't even switch browsers to get the functionality back since the others are even less configurable.

[1]: https://github.com/techwolfy/rainfox-theme

[2]: https://bugzilla.mozilla.org/show_bug.cgi?id=1358058

Luckily, I can still type "make install" without debian informing me that "random_dangerous_untrusted_code_from_interwebs" is not approved.

The cynical side of me says that it must not have this feature because if it did I'd have seen someone complaining about the browser "phoning home" or "forcing Mozilla's opinions into my eyeballs".

https://discourse.mozilla.org/t/certificate-issue-causing-ad...

    function set_xpi_sign_time_now() {
        const {Services} =  ChromeUtils.import("resource://gre/modules/Services.jsm");
        const now = (new Date()).getTime() / 1000;
        Services.prefs.setIntPref('app.update.lastUpdateTime.xpi-signature-verification', now);
    }
    
    set_xpi_sign_time_now();

This does the equivalent of setting in about:config the time of last signature verification to the current time. By default, Firefox re-checks signatures in 24 hours (or so I read somewhere here). I like the temporary effect of this, compared to the permanent disabling of signature verification suggested elsewhere.

----

1: https://developer.mozilla.org/en-US/docs/Tools/Browser_Conso...

as for a system to push messages to firefox users, is there anything like this in place? a standard? if so could it notify waterfox or icecat etc users at the same time as firefox users?

if there is no specification for this, is there a similar floss project that could be forked and molded into that of which we’re in need?

edit: nope, you cannot:(

[1] https://tools.ietf.org/html/rfc8555#section-7.6