Shai-Hulud malware attack: Tinycolor and over 40 NPM packages compromised

The general solution is to do what Debian does.

Keep a stable distro where new packages aren't added and versions change rarely (security updates and bugfixes only, no new functionality). This is what most people use.

Keep a testing/unstable distro where new packages and new versions can be added, but even then added only by the distro maintainer, NOT by the package developers. This is where the audits happen.

NPM, Python, Rust, Go, Ruby all suffer from this problem, because they have centralized and open package repositories.

Benefit from this feature.

Unfortunately most people don't want old software that doesn't support newer hardware so most people don't end up using Debian stable.

At the end of the day, it’s all a URL.

You’re asking for a blessed set of URLs. You’d have to convince someone to spend time maintaining that.

Things like: Once a package has more than [threshold] daily downloads for an extended period of time, it requires 2FA re-auth/step-up on two separate human-controlled accounts to approve any further code updates.

Or something like: for these popular packages, only a select list of automated build systems with reproducible builds can push directly to NPM, which would mean that any malware injector would need to first compromise the source code repository. Which, to be fair, wouldn't necessarily have stopped this worm from propagating entirely, but would have slowed its progress considerably.

This isn't a "sacrifice all of NPM's DX and decentralization" question. This is "a marginally more manual DX only when you're at a scale where you should be release-managing anyways."

Just wondering: while this is less of an attack surface, it's still a surface?

Can you point me to Go's centralized package repository?

Kinda like Stackoverflow for reviews, with optional identification and such.

And honestly an LLM can strap a "probably good" badge on things with cheap batch inference.

Someone could pony up the cash to send out a few thousand yubikeys for this and we'd all be a lot safer.

For example, an upstream bumps a version of a lint tool and/or changes style across the board. Often these are labelled "chore". While I agree it's nice to have consistent style, in some projects it seems to be the majority of the changes between releases. Due to the difficulty in auditing this, I consider this part of the software supply chain problem and something to be discouraged. Unless there's actually reason to change code (eg. some genuine refactoring a human thinks is actually needed, a bug fix or new feature, a tool exposed a real bug, or at least some identifiable issue that might turn into a bug), it should be left alone.

Or is this just a "don't use Linux" gripe?

Most people don't want old software because they don't want old software.

They want latest features, fixes and performance improvements.

https://difftastic.wilfred.me.uk/

If you ask these people, distributions are terrible and need to die.

Python even removed PGP signatures from Pypi because now attestation happens by microsoft signing your build on the github CI and uploading it directly to pypi with a never expiring token. And that's secure, as opposed to the developer uploading locally from their machine.

In theory it's secure because you see what's going in there on git, but in practice github actions are completely insecure so malware has been uploaded this way already.

Except most projects have 1 developer… Plus, if I develop some project for free I don't want to be wasting time and work for free for large rich companies. They can pay up for code reviews and similar things instead of adding burden to developers!

(I did a talk at minidebconf last year in toulouse about this).

If implemented like this, it's completely useless, since there is actually no 2fa at all.

Anyway the idea of making libre software developers work more is a bad idea. We do it for fun. If we have to do corporate stuff we want a corporate salary to go with.

If we want decentralized package management for node/javascript, you need to dump NPM - why not something like Go's system which is actually decentralized? There is no package repository/registry, it's all location based imports.

Edit: also, consider how much of https://github.com/Wilfred/difftastic/commits/master/ is just noise in itself. 15k commits for a project that appears to only be about four years old.

Adding friction to the sharing of code doesn't absolve developers from their decision to blindly trust a ridiculous amount of third-parties.

One idea I've had is that publishing is open as today, but security firms could offer audit signatures.

So a company might pay security firms and only accept updates to packages that have been audited by by 1,2,3 or more of their paid services.

Thus money would be paid in the open to have eyes on changes for popular packages and avoid the problem of that weird lone maintainer in northern Finland being attacked by the Chinese state.

> Unless there's actually reason to change code (eg. some genuine refactoring a human thinks is actually needed, a bug fix or new feature, a tool exposed a real bug, or at least some identifiable issue that might turn into a bug), it should be left alone.

The corollary to this is "Unless there's actually a need for new features that a new version provides, your existing dependency should be left alone". In other words things should not be automatically updated. This is unfortunately the crazy path we've gone down, where when Package X decides to upgrade, everyone believes that "the right thing to do" is for all its dependencies to also update to use that and so on down the line. As this snowballs it becomes difficult for any individual projects to hold the line and try to maintain a slow-moving, stable version of anything.

Which works together with

https://proxy.golang.org/

They collect package managers like funko pops.

I'm not quite sure about the goal. Maybe some more C# dev kit style rug-pulls where the ecosystem is nominally open-source but MS own the development and distribution so nobody would bother to compete.

The problem comes when you want to upgrade your dependencies. How do you know that they are trustworthy on first use?

If packages had to be cryptographically signed by multiple verified authors from a per-organization whitelist in order to enter distribution, that would cut down on the SPOF issue where compromising a single dev is enough to publish multiple malware-infested packages.

For what it's worth, our code is on GitLab

This is what happens when nobody pays for anything and nobody feels they have a duty to do good work for free.

1. Using an operating system with no package management 2. Poor developer discipline, i.e. developers always trying to use the latest version of a package.

So now we have lots of poorly implemented language package managers, docker containers on top being used as another package management layer (even though that's not their primary purpose but many people use the like that) and the security implications of pulling in lots of random dependencies without any audit.

Developing towards a stable base like Debian would not be a pancea, but alliviate the problems by at least placing another audit layer in between.

While the crates ecosystem is certainly not immune to supply chain attacks this over generalization is not justified.

There are several features that make crates.io more robust than npm. One of them is that vulnerable versions can be yanked without human intervention. Desperate comments from maintainers like this one[1] from just a few days ago would not happen with crates.io.

There are also features not provided by crates.io that make the situation better. For example you could very easily clone the repo and run

    cargo vet

[1] https://news.ycombinator.com/item?id=45170687

1. You don't want to tie your software to the OS. Most people want their software to be cross-platform. Much better to have a language-specific package manager because I'm using the same language on every OS. And when I say "OS" here, I really mean OS or Linux distro, because Linux doesn't have one package manager.

2. OS package managers (where they even exist), have too high a bar of entry. Not only do you have to make a load of different packages for different OSes and distros, but you have to convince all of them to accept them. Waaay too much work for all but the largest projects.

You're probably going to say "Good! It would solve this problem!", but I don't think the solution to package security is to just make it so annoying nobody bothers. We can do better than that.

For what it's worth, back when I was active at the ASF we used to vote on releases — you needed at least 3 positive votes from a whitelist of approved voters to publish a release outside the org and there was a cultural expectation of review. (Dunno if things have changed.) It would have been very difficult to duplicate this NPM attack against the upstream ASF release distribution system.

Weirdly, some of the worst CVE I can think of were with enterprize software.

And true 2FA means you can't automate publishing from github's CI. Python is going the other direction. There is a fake 2FA that is just used to generate tokens and there is a preferential channel to upload to pypi via github's CI.

But in my opinion none of this helps with security. But it does help to de-anonymise the developers, which is probably what they really want to do, without caring if those developers get hacked and someone else uses their identity to do uploads.

I do not know about NPM. But in Rust this is common practice.

Very hard to avoid. The core of Rust is very thin, to get anything done typically involves dozens of crates, all pulled in at compile time from any old developer implicitly trusted.

https://github.com/tursodatabase/turso/network/dependencies

"Large numbers of trustworthy dependency authors in your town can't wait to show you their hottest code paths! Click here for educational livecoding sessions!"

At this rate, there's a non-zero chance that one of the transitive dependencies is SQLite itself.

The reason these compromised packages typically don't make it in to e.g. Debian is because this all tends to be discovered quite quickly, before the package maintainer has a chance to update it.

Even if we make every project create packages in every package manager, it still wouldn't add any auditing.

I don't know how you avoid this problem

Vulnerabilities in Linux distro packages obviously happen. But a single developer cannot push code directly into for example Debian and compromise the world.

"old" is a strange way to spell "new, unstable, and wormed".

I want old software. Very little new features are added to most things i care about, mostly it is just bloat, AI slop, and monthly subscription shakedowns being added to software today.

Linux distros can't even provide all the apps users want, that's why freshmeat existed and we have linuxbrew, flatpak, Ubuntu multiverse, PPA, third party Debian repositories, the openSUSE Buildservice, the AUR, ...

There is no community that has the capacity to audit and support multiple branches of libraries.

Establishing a false identity well enough to fool a FOSS author or organization is a lot of work. Even crafting a spear phishing email/text campaign doesn't compare to the effort you'd have to put in to fool a developer well enough to get offered publishing privileges.

Of course it's possible, but so are beat-them-with-a-five-dollar-wrench attacks.

However we are talking in the context of NPM packages which by the vast majority would be running inside a container on some server. So how could that software not use a stable Debian base for example.

And arguing that package management is to complicated is a bit ridiculous considering how many workloads are running in docker containers which I'd argue are significantly more complex

Those numbers are way off their actual number.

https://lintian.debian.org/

https://github.com/crev-dev/

https://github.com/crev-dev/

How can you guarantee a long trusted developer doesn't have a gun pointed to their head by their authoritarian govt?

In our B2B shop we recently implemented a process where developers cannot add packages from third party sources - only first party like meta, google, spring, etc are allowed. All other boilerplate must be written by developers, and on the rare occasion that a third party dependency is needed it's copied in source form, audited and re-hosted on our internal infrastructure with an internal name.

To justify it to business folks, we presented a simple math where I added the man-hours required to plug the vulnerabilities with the recurring cost of devsecops consultants and found that it's cheaper to reduce development velocity by 20-25%.

Also devsecops should never be offshored due to the scenario I presented in my second statement.

In both JavaScript and Rust, it's normal/encouraged to just add a tiny dependency to the package manager. The communities even pride themselves, that they have such good package managers to allow this.

It's this "yeah, there is a crate for this tiny function I need, let's just include it" mentality that makes the ecosystem vulnerable.

People need to be responsible for whatever they include, either pay the price by checking all versions up front, or pay it by risking shipping a vulnerable program that it's much harder to retract than a JavaScript frontend.

* You are trusting large numbers of trustworthy developers.

* You have established a means of validating their trustworthiness: only trust reputable "first-party" code.

I think what you're doing is a pretty good system. However, there are ways to include work by devs who lack "first-party" bona-fides, such as when they participate in group development where their contributions are consistently audited. Do you exclude packages published by the ASF because some contributions may originate from troublesome jurisdictions?

In any case, it is not necessary to solve the traitorous author problem to address the attack vector right in front of us, which is compromised authors.

That said, I would guess the 'bigger conversation' is that it is much harder to tpyo <<import "github.com/DataaDog/datadog-api-client-go/v2/api/datadogV2">> than $(npm i dataadog) or similar in a "flat" package namespace (same for its $(uv pip install dataadog) friend)

None of those cited ones fix the dependency lineage issue, proving that release 1.1 was authored by the same chain of custody as release 1.0 of any given package. One can opt in to gpg verified dependencies in Maven, but it is opt-in. The .jar artifacts can also be cryptographically signed, but the risk that's trying to drive down is tamperproofing and not lineage, AFAIK

Then what's the difference between git and npm, cargo, pypi, mvn et al?

FWIW npm used to allow unpublishing packages, but AFAIK that feature was removed in the wake of the left-pad incident [1]. Altho now with all the frequent attacks, it might be worth considering if ecosystem disruption via malicious removal of pacakge would be lesser of two evils, compared to actual malware being distributed.

1: https://en.wikipedia.org/wiki/Npm_left-pad_incident

Linux distros could still stand to improve here in a bunch of ways, and it seems that a well-designed package ecosystem truly doesn't need such hooks at the level of the package manager at all. But this kind of auditing is one of the useful functions of downstream software distros for sure.

That JLR got their factories hacked, rather than customer cars, is less bad for sure. But it's still pretty bad.

Also, before arguing that code generators should get a pass as they don't “end up in the final product”, you really should read “Reflections on trusting trust” by Ken Thompson.

Most projects will have a healthy 5-20 dependencies though, with very little nested modules.

That's bullshit, pure and simple. If you pull in a deeply nested dependency like icu_normalizer it has 30 dependencies, OMGHAXOZRS. I'm doing this, so I don't have to spend a day going through the library.

Except of the 30 depedencies crates, there are 10 from ICUX repository, and then you have almost standard dependencies like proc-macro/syn/quote crates from dtolnay, `zerofrom` from Google. `smallvec` from the Servo project, and yoke from... checks notes... from ICUX.

The only few remaining crates here are `write16`, `utf8_iter` and `utf16_iter` that are written from hsivonen, who is also a ICUX contributor.

So even for 30 dependencies, you actually depend on proc-macro/syn/quote which are foundational crates. Few crates from Google, few crates from Servo, and three crates written by another ICUX contributor.

We started with 30 dependencies and ended up with 3 strangers

The problem with this approach is that frameworks tend to "expire" pretty quickly and you can't run anything for too long on Debian until the framework is obsolete. What I mean by obsolete is Debian 13 ships with Golang 1.24, A year from now it's gonna be Golang 1.26 - that is not being made available in trixie. So you have to find an alternative source for the latest golang deb. Same with PHP, Python etc. If you run them for 3 years with no updated just some security fixes here and there, you're gonna wake up in a world of hurt when the next stable release comes out and you have to do en-masse updates that will most likely require huge refactoring because syntax, library changes and so on.

And Javascript is a problem all by itself where versions come up every few months and packages are updated weekly or monthly. You can't run any "modern" app with old packages unless you accept all the bugs or you put in the work and fix them.

I am super interested in a solution for this that provides some security for packages pushed to NPM (the most problematic repository). And for distributions to have a healthy updated ecosystem of packages so you don't get stuck who knows for how long on an old version of some package.

And back to Debian, trixie ships with nginx 1.26.3-3+deb13u1. Why can't they continuously ship the latest stable version if they don't want to use the mainline one?

But how do we scale that to 1000 dependencies, and every one of their updates? What tools are there to help us, and does the community at large use them?

What I really don't like, and why I wrote that it's a culture issue, is the lightness with which these decisions are often made.

My most popular library has about a dozen dependencies. The README states clearly and “above the fold” what are the core deps (3, no transitive). Every other dependency is either first party, or optional and justified with a comment in the deps file (if you don't use the optional feature, it doesn't end up in your deps file).

There's also a generated BLOB. The generation of the BLOB is reproducible in your own environment, and its provenance attestated.

Those are all risks, that I'm passing on to my users, but I do my best to mitigate them, and communicate this clearly to them.

In practice, little difference between Go's use of Github and Python's use of PyPI. Someone at Microsoft with root access could compromise everyone.

Isn't it actually the case that you started with 3 strangers, but 27 of them were relatively easy (still took some time) to figure out as safe?

If you add jest, the popular test runner by Meta, that's adding 300 packages to your dependency graph.

And here we don't yet have a bundler, linter, code formatter, or even web framework.

So good luck with minimizing those dependencies.

You could of course investigate individuals commits, but that's probably an overkill.

That's why I'm putting emphasis on it, because to Go it is.

And to languages that actually have centralized package repositories it isn't. There is a difference between code and packages and Go simply does not have the latter (in the traditional sense - what Go calls a package is a collection of source files in the same directory that are compiled together within a module (a module is a collection of packages (again, code) that are released, versioned, and distributed together. Modules may be downloaded directly from version control repositories or via proxy servers)).

To the other languages mentioned above, packages may have binaries, metadata and special script hooks. There is a package manager like pip , cargo or npm and if you want to install one, you won't have to specify a URL because there is a canonical domain to go to.

Go just knows code and it'll use git, hg or even svn. And if you want to claim that lots of open-source code being on GitHub makes it special, then

> GitHub is every single programming language's centralized package repository

and

> Someone at Microsoft with root access could compromise every user of every single programming language

95% of Python packages are installed from PyPI, but just like Go can also install from non-Github sources, Python supports installing from other non PyPI indexes[0] or even from a Git repository directly[1] like Go.

> what Go calls a package is a collection of source files in the same directory

What is it that you imagine Python or NPM packages consist of? Hint: A Python .whl file is just a folder in a zip archive (Python also supports source distributions directly analogous to Go)

[0] https://docs.astral.sh/uv/concepts/indexes/

[1] https://thelinuxcode.com/install-git-repository-branch-using...

So "GitHub is every single programming language's centralized package repository, because lots of code is hosted there" ?

> Python supports installing from other non PyPI indexes > 95% of Python packages are installed from PyPI, but just like Go can also install from non-Github sources, Python supports installing from other non PyPI indexes[0] or even from a Git repository directly[1] like Go.

And yet there is a clear difference between source distributions and pip/npm/rubygem/cargo packages - and between tooling/ecosystems that ONLY support the former and those that MAY use either and unfortunately mostly use the latter.

> What is it that you imagine Python or NPM packages consist of?

Something like a script that runs as part of the package that downloads a tarball, modifies package.json, injects a local bundle.js and runs npm publish (see this post). Usually also hosted at the default, centralized, authoritative source run by the maintainers of the package management tool.

But I'm repeating myself.

> (or do they? do they just use https to check out?)

Maybe try it out or read the docs first.

I'm closing with this:

> NPM, Python, Rust, Go, Ruby all suffer from this problem, because they have centralized and open package repositories.

is either wrong or disingenuously misleading, requiring nothing to apply to every single thing, depending on how you slice your definitions. It does not hold any water, that is my entire argument.