Most active commenters
  • DoctorOetker(4)

←back to thread

All extensions disabled due to expiration of intermediate signing cert

(bugzilla.mozilla.org)
1318 points xvector | 17 comments | 04 May 19 01:31 UTC | HN request time: 1.325s | source | bottom
Show context
rmbryan ◴[04 May 19 10:27 UTC] No.19825581[source]▶
Update: We have rolled out a partial fix for this issue. We generated a new intermediate certificate with the same name/key but an updated validity window and pushed it out to users via Normandy (this should be most users). Users who have Normandy on should see their add-ons start working over the next few hours. We are continuing to work on packaging up the new certificate for users who have Normandy disabled.
replies(20): >>19825596 #>>19825603 #>>19825612 #>>19825623 #>>19825631 #>>19825665 #>>19825705 #>>19825721 #>>19825744 #>>19825813 #>>19825905 #>>19825998 #>>19826421 #>>19826769 #>>19826772 #>>19826878 #>>19827050 #>>19829585 #>>19831941 #>>19840386 #
DoctorOetker ◴[04 May 19 11:00 UTC] No.19825705[source]▶
I read at https://discourse.mozilla.org/t/certificate-issue-causing-ad...

>12:50 p.m. UTC / 03:50 a.m. PDT: We rolled-out a fix for release, beta and nightly users. The fix will be automatically applied in the background within the next few hours, you don’t need to take active steps.

>In order to be able to provide this fix on short notice, we are using the Studies system. You can check if you have studies enabled by going to Firefox Preferences -> Privacy & Security -> Allow Firefox to install and run studies.

>You can disable studies again after your add-ons have been re-enabled.

>We are working on a general fix that doesn’t need to rely on this and will keep you updated.

I refuse to enable studies, even temporarily. This comes very close after the IE6 conspiracy revelation, where ends justifies the means.

Please provide a link to the certificate file, and step by step instructions for installing it, without enabling and conflating with mozilla studies...

replies(3): >>19825894 #>>19825895 #>>19825921 #
1. XORcat ◴[04 May 19 12:04 UTC] No.19825921[source]▶
JSON response from the `normandy` API here: https://xor.cat/assets/other/random/2019-05-04/normandy_sign...

hotfix-update-xpi-signing-intermediate-bug-1548973: https://storage.googleapis.com/moz-fx-normandy-prod-addons/e...

From the looks, it installs the above plugin, and changes `app.update.lastUpdateTime.xpi-signature-verification` to `1556945257`

I can't get it to work in ESR 60 though. Getting file not found on "resource://gre/modules/addons/XPIDatabase.jsm"

edit: The linked XPI definitely seems to add the new certificate, whatever mechanism used to reverify the signatures just doesn't seem to work in 60.

edit2: Restarting Firefox appears to have forced the reverify... Possibly a flag that I twiddled with though, hard to be sure. Either way, the above should help people get everything running again without having to enable studies/normandy.

replies(5): >>19826115 #>>19826755 #>>19827210 #>>19827215 #>>19827221 #
2. rndgermandude ◴[04 May 19 12:54 UTC] No.19826115[source]▶
Yes, this is broken on ESR, but only somewhat broken.

The hotfix extension does two things:

1) Install a new certificate for "CN=signingca1.addons.mozilla.org/emailAddress=foxsec@mozilla.com", effectively replacing the old certificate that expired. This should work.

2) Then it tries to import the internal "resource://gre/modules/addons/XPIDatabase.jsm" module and calls XPIDatabase.verifySignatures().

This does not work on ESR, as "XPIDatabase.jsm" is a new-ish thing that isn't present in ESR yet. In ESR the function is still in "resource://gre/modules/addons/XPIProvider.jsm" (XPIProvider.verifySignatures()). Thankfully, the non-existing module is imported using ChromeUtils.defineModuleGetter, which only lazily loads the module on first of the imported property, so after the certificate-adding code has run.

3. gpm ◴[04 May 19 14:45 UTC] No.19826755[source]▶
Hey, if you just click on that storage.googleapis.com link it installs the hotfix directly without having to enable normandy ;)
replies(6): >>19826795 #>>19826847 #>>19826887 #>>19827014 #>>19827736 #>>19828312 #
4. option_greek ◴[04 May 19 14:54 UTC] No.19826795[source]▶
This should be sticky comment somewhere on the top of the comments. It bought all the addons back for me.
5. jwalton ◴[04 May 19 15:02 UTC] No.19826847[source]▶
It does, but it didn't fix anything for me. All my extensions are still gone. :(
replies(1): >>19826864 #
6. gpm ◴[04 May 19 15:05 UTC] No.19826864{3}[source]▶
You might have to reinstall them unfortunately, on the system I figured that out on Firefox had decided to uninstall them (I think because I had to update the browser from the ancient version the user was using first).
7. ◴[04 May 19 15:09 UTC] No.19826887[source]▶
8. classichasclass ◴[04 May 19 15:25 UTC] No.19827014[source]▶
Just tried on Android. Hooray!
replies(1): >>19827192 #
9. mateus1 ◴[04 May 19 15:52 UTC] No.19827192{3}[source]▶
Clicking the URL was the only way I was able to get the hotfix on Firefox mobile for Android
10. 1over137 ◴[04 May 19 15:55 UTC] No.19827210[source]▶
So not only does this 'normandy' thing exist, but it goes to a google server? So much for using Firefox to keep google out of my life. :(
replies(1): >>19827771 #
11. johnnycab ◴[04 May 19 15:56 UTC] No.19827215[source]▶
>hotfix-update-xpi-signing-intermediate-bug-1548973: https://storage.googleapis.com/moz-fx-normandy-prod-addons/e...

This fixed it for me. Thanks. W10/FF 66.0.3

12. DoctorOetker ◴[04 May 19 15:57 UTC] No.19827221[source]▶
Thanks for the sleuthing, but who does this repository belong to? I'd like to apply it but only if mozilla provides such instruction on their issue page, I don't know who the actual owner of /moz-fx-normandy-prod-addons/ is...

https://storage.googleapis.com/moz-fx-normandy-prod-addons/e...

Can mozilla please verify, confirm authenticity, and list this instruction on their issue page?

replies(1): >>19830574 #
13. reader_1000 ◴[04 May 19 17:07 UTC] No.19827736[source]▶
Unrelated to cert problem: Yes, clicking on the link installs the plugin, but it is suprising to see that firefox claims that it is the news.ycombinator.com, not storage.googleapis.com, that wants to install plugin. Could it be a security issue since if an attacker somehow manages the post/inject a link for a malicious plugin in a credible site, firefox will claim that plugin is from that site?
replies(1): >>19827781 #
14. DoctorOetker ◴[04 May 19 17:13 UTC] No.19827771[source]▶
that's an interesting question: when we install add-ons or extensions, are these hosted on google servers? I'd rather not have google know what versions of which add-ons I am running...
15. DoctorOetker ◴[04 May 19 17:15 UTC] No.19827781{3}[source]▶
oh wow! that's really bad
16. devcpp ◴[04 May 19 18:35 UTC] No.19828312[source]▶
I just installed some random googleapis link. This is so stupid, and very disappointing from Mozilla.
17. XORcat ◴[05 May 19 01:43 UTC] No.19830574[source]▶
I would have the same question if I didn't see the response come back from https://normandy.cdn.mozilla.net/ myself.

I encourage you to go through the whole Normandy process yourself in a test environment, and even better (if possible), check out the code to see whether it looks legit or benign.

I'm happy, because I went through and checked it out myself without needing to enable Normandy on my actual Firefox, but ultimately, it will be great when Moz can get instructions for manually applying the fix out.