Shai-Hulud malware attack: Tinycolor and over 40 NPM packages compromised

This is a culture issue with developers who find it OK to have hundreds of (transitive) dependencies, and then follow processes that, for all intents and purposes, blindly auto update them, thereby giving hundreds of third-parties access to their build (or worse) execution environments.

Adding friction to the sharing of code doesn't absolve developers from their decision to blindly trust a ridiculous amount of third-parties.

Unfortunately that's almost the whole industry. Every software project I've seen has an uncountable amount of dependencies. No matter if npm, cargo, go packages, whatever you name.

I find that the issue is much more often not updating dependencies often enough with known security holes, than updating too often and getting hit with a supply-chain malware attack.

Be that as it may, a system that can fail catastrophically will. Security shouldn't be left to choice.

It's not unreasonable to trust large numbers of trustworthy dependency authors. What we lack are the institutions to establish trust reliably.

If packages had to be cryptographically signed by multiple verified authors from a per-organization whitelist in order to enter distribution, that would cut down on the SPOF issue where compromising a single dev is enough to publish multiple malware-infested packages.

Every place I ever worked at made sure to curate the dependencies for their main projects. Heck, in some cases that was even necessary for certifications. Web dev might be a wild west, but as soon as your software is installed on prem by hundreds or thousands of paying customers the stakes change.

Curating dependencies won't prevent all supply chain attacks though

Rather than adding friction there is something else that could benefit from having as little friction as sharing code: publishing audits/reviews.

There have been several recent supply chain attacks that show attackers are taking advantage of this (previously sensible) mentality. So it is time to pivot and come up with better solutions before it spirals out of control.

Not updating is the other side of the same problem: library owners feel it is ok to make frequent backwards-compatibility breaking changes, often ignoring semver conventions. So consumers of their libraries are left with the choice to pin old insecure versions or spend time rewriting their code (and often transitive dependency code too) to keep up.

This is what happens when nobody pays for anything and nobody feels they have a duty to do good work for free.

Problem is that beyond some threshold number of authors, the probability they're all trustworthy falls to zero.

It's true that smuggling multiple identities into the whitelist is one attack vector, and one reason why I said "cut down" rather than "eliminate". But that's not easy to do for most organizations.

For what it's worth, back when I was active at the ASF we used to vote on releases — you needed at least 3 positive votes from a whitelist of approved voters to publish a release outside the org and there was a cultural expectation of review. (Dunno if things have changed.) It would have been very difficult to duplicate this NPM attack against the upstream ASF release distribution system.

Zero-external-dependency Go apps are far more feasible than Rust or Node, simply because of the size and quality of the standard library.

>This is what happens when nobody pays for anything and nobody feels they have a duty to do good work for free.

Weirdly, some of the worst CVE I can think of were with enterprize software.

See auto update bots on Github. https://docs.github.com/en/code-security/dependabot/dependab... And since Github does it, it must be a good thing, right? Right???

> This is a culture issue with developers who find it OK to have hundreds of (transitive) dependencies, and then follow processes that, for all intents and purposes, blindly auto update them

I do not know about NPM. But in Rust this is common practice.

Very hard to avoid. The core of Rust is very thin, to get anything done typically involves dozens of crates, all pulled in at compile time from any old developer implicitly trusted.

That's because there many people don't feel like it is their duty to do good work, even though they are paid ...

Just the other day someone argued with me that it was reasonable for Limbo (the SQLite Rust rewrite) to have 3135 dependencies (of those, 1313 Rust dependencies).

https://github.com/tursodatabase/turso/network/dependencies

"Find large numbers of trustworthy dependency authors in your neighborhood!"

"Large numbers of trustworthy dependency authors in your town can't wait to show you their hottest code paths! Click here for educational livecoding sessions!"

This is incredible.

At this rate, there's a non-zero chance that one of the transitive dependencies is SQLite itself.

A model that Linux distros follow would work to an extent: you have developed of packages and separate maintainers who test and decide to include or exclude packages and versions of packages. Imagine a JS distro which includes the top 2000 most popular libraries that are all known to work with each other. Your project can pull in any of these and every package is cryptographically signed off on by both the developers and the maintainer.

Vulnerabilities in Linux distro packages obviously happen. But a single developer cannot push code directly into for example Debian and compromise the world.

I don't understand your critique.

Establishing a false identity well enough to fool a FOSS author or organization is a lot of work. Even crafting a spear phishing email/text campaign doesn't compare to the effort you'd have to put in to fool a developer well enough to get offered publishing privileges.

Of course it's possible, but so are beat-them-with-a-five-dollar-wrench attacks.

The same is true for go and for java.

Yeah. You have dev dependencies in there, those alone will increase number of dependencies by ~500, without ending up in the final product.

Those numbers are way off their actual number.

It IS unreasonable to trust individual humans across the globe in 100+ different jurisdictions pushing code that gets bundled into my application.

How can you guarantee a long trusted developer doesn't have a gun pointed to their head by their authoritarian govt?

In our B2B shop we recently implemented a process where developers cannot add packages from third party sources - only first party like meta, google, spring, etc are allowed. All other boilerplate must be written by developers, and on the rare occasion that a third party dependency is needed it's copied in source form, audited and re-hosted on our internal infrastructure with an internal name.

To justify it to business folks, we presented a simple math where I added the man-hours required to plug the vulnerabilities with the recurring cost of devsecops consultants and found that it's cheaper to reduce development velocity by 20-25%.

Also devsecops should never be offshored due to the scenario I presented in my second statement.

You've presented your argument as if rebutting mine, but to my mind you've reinforced my first paragraph:

* You are trusting large numbers of trustworthy developers.

* You have established a means of validating their trustworthiness: only trust reputable "first-party" code.

I think what you're doing is a pretty good system. However, there are ways to include work by devs who lack "first-party" bona-fides, such as when they participate in group development where their contributions are consistently audited. Do you exclude packages published by the ASF because some contributions may originate from troublesome jurisdictions?

In any case, it is not necessary to solve the traitorous author problem to address the attack vector right in front of us, which is compromised authors.

500 dev dependencies doesn’t seem reasonable either…

Even 50 seems unreasonable...

Right. Allowing 500 strangers to push code to our CI infra, or developer laptops, with approximately zero review, sounds similarly ill advised.

That JLR got their factories hacked, rather than customer cars, is less bad for sure. But it's still pretty bad.

Also, before arguing that code generators should get a pass as they don't “end up in the final product”, you really should read “Reflections on trusting trust” by Ken Thompson.

But it will be safe SQlite, called from Rust.

Even more wild considering that SQLite prides itself on having zero dependencies. Sounds like a doomed project.

You can write entire applications in Go without resorting to any dependencies, the std lib is quite complete.

Most projects will have a healthy 5-20 dependencies though, with very little nested modules.

> Right. Allowing 500 strangers to push code to our CI infra

That's bullshit, pure and simple. If you pull in a deeply nested dependency like icu_normalizer it has 30 dependencies, OMGHAXOZRS. I'm doing this, so I don't have to spend a day going through the library.

Except of the 30 depedencies crates, there are 10 from ICUX repository, and then you have almost standard dependencies like proc-macro/syn/quote crates from dtolnay, `zerofrom` from Google. `smallvec` from the Servo project, and yoke from... checks notes... from ICUX.

The only few remaining crates here are `write16`, `utf8_iter` and `utf16_iter` that are written from hsivonen, who is also a ICUX contributor.

So even for 30 dependencies, you actually depend on proc-macro/syn/quote which are foundational crates. Few crates from Google, few crates from Servo, and three crates written by another ICUX contributor.

We started with 30 dependencies and ended up with 3 strangers

Who do you mean with "many people"? Developers who do not care or middle management that oversold features and overcommitted w.r.t. deadlines? Or both? Someone else?

It's great that you did that due diligence once. It really is. If I were reviewing that merge request, and you told me that, I'd be inclined to approve.

But how do we scale that to 1000 dependencies, and every one of their updates? What tools are there to help us, and does the community at large use them?

What I really don't like, and why I wrote that it's a culture issue, is the lightness with which these decisions are often made.

My most popular library has about a dozen dependencies. The README states clearly and “above the fold” what are the core deps (3, no transitive). Every other dependency is either first party, or optional and justified with a comment in the deps file (if you don't use the optional feature, it doesn't end up in your deps file).

There's also a generated BLOB. The generation of the BLOB is reproducible in your own environment, and its provenance attestated.

Those are all risks, that I'm passing on to my users, but I do my best to mitigate them, and communicate this clearly to them.

Sorry, I don't know much about the subject, so this is not a rhetorical or even just loaded question:

Isn't it actually the case that you started with 3 strangers, but 27 of them were relatively easy (still took some time) to figure out as safe?

If someone is wondering how effective such an approach is going to be with npm, consider the following:

If you add jest, the popular test runner by Meta, that's adding 300 packages to your dependency graph.

And here we don't yet have a bundler, linter, code formatter, or even web framework.

So good luck with minimizing those dependencies.

I was thinking of many developers, but actually middle management should be included.

3 different types of sqlite, 14 different versions total: https://github.com/tursodatabase/turso/network/dependencies?...

You have 30 items you bought from various stores. You bought ~20 from yourself, around 5 from Google, 4 from hsivonen and one from servo.

You could of course investigate individuals commits, but that's probably an overkill.

Looks like they're all pulled in as dev dependencies. libsqlite3-sys gets pulled in by rusqlite, which is used by core_tester, limbo_sim, write-throughput-sqlite, and as a dev_dependency for turso_core.

> But how do we scale that to 1000 dependencies, and every one of their updates? What tools are there to help us, and does the community at large use them?

Use

    cargo install cargo-supply-chain
    cargo supply-chain --publishers

Run it for whatever you want to check, then have a lunch, it takes 10-30min.

It will list exactly how many organizations, and even individuals with publish rights are there. For turso there are 51 repositories, and 243 different individuals with publish rights.

Of course, this still doesn't group by github org and so on.