Shai-Hulud malware attack: Tinycolor and over 40 NPM packages compromised

As a security guy, for years, you get laughed out of the room suggesting devs limit their dependencies and don't download half of the internet while building. You are an obstruction for making profit. And obviously reading the code does very little since modern (and especially Javascript) code just glues together frameworks and libraries, and there's no way a single human being is going to read a couple million lines of code.

There are no real solutions to the problem, except for reducing exposure somewhat by limiting yourself to a mostly frozen subset of packages that are hopefully vetted more stringently by more people.

I've always been very careful about dependencies, and freezing them to versions that are known to work well.

I was shocked when I found out that at some of the most profitable shops, most of their code is just a bunch of different third-party libraries badly cobbled together, with only a superficial understanding of how those libraries work.

> You are an obstruction for making profit.

This explains a lot. Really, this is the great reason of why the society is collapsing as we speak.

"There should be no DRM in phones" - "You Are An Obstruction To Making Profit".

"People should own their devices, we must not disallow custom software on it" - "YAAOTMP"

"sir, the application will weigh 2G and do almost nothing yet, should we minify it or use different framework?" - "YAAOTMP".

"Madame, this product will cost too much and require unnecessary payments" - "YAAOTMP"

Etc. etc. Like in this "Silicon Valley" comedy series. But for real, and affecting us greatly.

At my previous enterprise we had a saying:

Security: we put the ‘no’ in ‘innovation’.

The "solution" would be using a language with a strong standard library and then having a trusted 3rd party manually audit any approved packages.

THEN use artifactory on top of that.

That's boring and slow though. Whatever I want my packages and I want them now. Apart of the issue is the whole industry is built upon goodwill and hope.

Some 19 year old hacked together a new front end framework last week, better use it in prod because why not.

Occasionally I want to turn off my brain and just buy some shoes. The Timberland website made that nearly impossible last week. When I gave up on logging in for free shipping and just paid full price, I get an email a few days later saying they ran out of shoes.

Alright. I guess Amazon is dominant for a reason.

This is the right answer. I'm willing to stick my head out and assert that languages with a "minimal" standard library are defective by design. The argument of APIs being stuck is mood with approaches like Rust's epocs or "strict mode".

Standard libraries should include everything needed to interact with modern systems. This means HTTP parsing, HTTP requests, and JSON parsing. Some laguages are excellent (like python), while some are half way there (like go), and some are just broken (Rust).

External libraries are for niche or specialized functionality. External libraries are not for functionality that is used by most modern software. To put your head in the ground and insist otherwise is madness and will lead to ridiculous outcomes like this.

I don’t recall hearing about constant supply chain attacks with CPAN

I agree, it always seems to be NPM, and there's a reason for that.

Because it's never been considered an interesting target, compared to npm's reach?

Java + Spring Boot BOM + Maven Central (signed jars) does fit the description.

> External libraries are not for functionality that is used by most modern software.

Where do you draw the line though? It seems like you mostly spend your time writing HTTP servers reading/writing JSON, but is that what everyone else also spends their time doing? You'll end up with a standard library weighing GBs, just because "most developers write HTTP servers", which doesn't sound like a better solution.

I'm willing to stick my head the other way, and say I think the languages today are too large. Instead, they should have a smaller core, and the language designed in a way that you can extend the language via libraries. Basically more languages should be inspired by Lisps and everything should be a library.

Death comes to corp CEO, he screams YAAOTMP, death leaves shocked. Startup CEO watches the scene. His jedi sword turns from blue to red.

> Standard libraries should include everything needed to interact with modern systems.

So, databases? Which then begs the question, which - Postgres, MySQL, SQLite, MS SQL, etc.? And some NoSQL, because modern systems might need it.

That basically means you need to pull in everything and the kitchen sink. And freeze it in time (because of backwards compatibility). HTML, HTTP parsing, and SHA1024 are perfectly reasonable now; wait two decades, and they might be as antiquated as XML.

So what your language designers end up, is having to work on XML parsing, HTTP, JSON libraries rather than designing a language.

If JS way is madness, having everything available is another form of madness.

> Standard libraries should include everything needed to interact with modern systems.

This is great when the stdlib is well-designed and kept current when new standards and so on become available, but often "batteries included" approaches fail to cover all needs adequately, are slow to adopt new standards or introduce poorly designed modules that then cannot be easily changed, and/or fail to keep up-to-date with the evolution of the language.

I think the best approach is to have a stdlib of a size that can be adequately maintained/improved, then bless a number of externally developed libraries (maybe even making them available in some official "community" module or something with weaker stability guarantees than the stdlib).

I find it a bit funny that you specifically say HTTP handling and JSON are the elements required when that's only a small subset of things needed for modern systems. For instance, cryptography is something that's frequently required, and built-in modules for it often suck and are just ignored in favor of external libraries.

EDIT: actually, I think my biggest issue with what you've said is that you're comparing Python, Go, and Rust. These languages all have vastly different design considerations. In a language like Python, you basically want to be able to just bash together some code quickly that can get things working. While I might dislike it, a "batteries included" approach makes sense here. Go is somewhat similar since it's designed to take someone from no knowledge of the language to productive quickly. Including a lot in the stdlib makes sense here since it's easier to find stuff that way. While Rust can be used like Python and Go, that's not really its main purpose. It's really meant as an alternative to C++ and the various niches C/C++ have dominated for years. In a language like that, where performance is often key, I'd rather have a higher quality external library than just something shoved into the stdlib.

I don't think things being libraries (modular) is at odds with a standard library.

If you have a well vetted base library, that is frequently reviewed, under goes regular security and quality checks, then you should be minimally concerned about the quality of code that goes on top.

In a well designed language, you can still export just what you need, or even replace parts of that standard library if you so choose.

This approach even handles your question: as use cases become more common, an active, invested* community (either paying or actively contributing) can add and vet modules, or remove old ones that no longer serve an active purpose.

But as soon as you find yourself "downloading the web" to get stuff done, something has probably gone horribly wrong.

That was a different era. The velocity of change is 100x now and the expectation for public libraries to do common things is 100x higher as well.

Java is around for much longer, has exactly same architecture re transitive dependencies, yet doesn't suffer from weekly attacks like these that affect half of the world. Not technically impossible, yet not happening (at least not at this scale).

If you want an actual solution, look for differences. If you somehow end up figuring out its about type of people using those, then there is no easy technical solution.

This comes across as not being self-aware as to why security as laughed out of rooms: I read this as you correctly identifying some risks and said only offered the false-dichotomouy of solutions of "risk" and "no risk" without talking middle grounds between the two or finding third-ways that break the dichotomy.

I could just be projecting my own bad experiences with "security" folks (in quotes as I can't speak to their qualifications). My other big gripe is when they don't recongnize UX as a vital part of security (if their solution is unsuable, it won't be used).

> This is the right answer. I'm willing to stick my head out and assert that languages with a "minimal" standard library are defective by design.

> Standard libraries should include everything needed to interact with modern systems. This means HTTP parsing, HTTP requests, and JSON parsing.

There is another way. Why not make the standard library itself pluggable? Rust has a standard library and a core library. The standard library is optional, especially for bare-metal targets.

Make the core library as light as possible, with just enough functionality to implement other libraries, including the interfaces/shims for absolutely necessary modules like allocators and basic data structures like vectors, hashmaps, etc. Then move all other stuff into the standard library. The official standard library can be minimal like the Rust standard library is now. However, we should be able to replace the official standard library with a 3rd party standard library of choice. (What I mean by standard library here is the 'base library', not the official library.) Third party standard library can be as light or as comprehensive as you might want. That also will make auditing the default codebase possible.

I don't know how realistic this is, but something similar is already there in Rust. While Rust has language features that support async programming, the actual implementation is in an external runtime like Tokio or smol. The clever bit here is that the other third party async libraries don't enforce or restrict your choice of the async runtime. The application developer can still choose whatever async runtime they want. Similarly, the 3rd party standard library must not restrict the choice of standard libraries. That means adding some interfaces in the core, as mentioned earlier.

IMO Python 2 was rhetorical gold standard for getting the std lib right. Mostly batteries included, but not going totally insane with it.

The tradeoff of “batteries included” vs not is real: Python developers famously reach for community libraries like requests right away to avoid using the built-in tooling.

I've been a web developer for over two decades. I have specific well-tested solutions for avoiding external JS dependencies. Despite that, I have the exact same experience as the above security guy. Most developers love adding dependencies.

The post you replied to suggested a real solution to the problem. It was implemented in my current org years ago (after log4j) and we have not been affected by any of the malware dependencies that has happened since.

Perl and CPAN are still a thing, much as people would like to think otherwise.

Your proposed solution does not work for web applications built with node packages.

Essentials tools such as Jest add 300 packages on their own.

You already have hundreds to thousands of packages installed, fretting over a few more for that DatePicker or something is pretty much a waste of time.

It's not an easy problem to solve.

Doing it the right way would create friction, developers might need to actually understand what the code is doing rather than pulling in random libraries.

Try explaining to your CTO that development will slow down to verify the entire dependency chain.

I'm more thinking C# or Java. If Microsoft or Oracle is providing a library you can hope it's safe.

You *could* have a development ecosystem called Safe C# which only comes with vetted libraries and doesn't allow anything else.

I'm sure other solutions already exist though.

I wasn't even aware there _was_ built-in tooling...

It is not madness. Java is a good example of rich and modular standard library. Some components of it are eventually deprecated and removed (e.g. Applets) and this process takes long enough. Its standard library does include good crypto and http client, database abstraction API (JDBC) which is implemented by database drivers etc.

Spot on, I rather have a Python, Java,.NET,.. standard library, that may have a few warts, but works everywhere there is full compliant implementation, than playing lego, with libraries that might not even support all platforms, and be more easily open to such attacks.

Is java.util.logging.Logger not that great?

Sure, yet everyone that used it had a good night rest when Log4J exploit came to be.

Why?

This is a standard practice in most places I have worked, CI/CD only allowed to use internal repos, and libraries are only added after clearance.

Agree on the only solution being reducing dependencies.

Even more weird in the EU where things like Cyber Resilience Act mandate patching publicly known vulnerabilities. Cool, so let's just stay up2date? Supply-chain vuln goes Brrrrrr

Yeah, and Java was always corporately funded, and to my knowledge no one really used neither the http client nor the XML parser. You basically have a collection of dead weight libs, that people have to begrudgingly maintain.

Granted some (JDBC) more useful than the others. Although JDBC is more of an API and less of a library.

My favourite is java.awt.Robot

HttpClient is relatively new and getting HTTP/3 support next spring, so it’s certainly not falling into the dead weight category. You are probably confusing it with an older version from Java 1.1/1.4.

As for XML, JAXP was a common way to deal with it. Yes, there’s Xstream etc, but it doesn’t mean any of standard XML APIs are obsolete.

Except that "clearance" invariably consists of bureaucratic rubber stamping and actually decreases security by making it harder and slower to fix newly discovered vulnerabilities.

Depends on the skills of the respective DevOps security team.

There are also tools that break CI/CD based on CVE reports from existing dependencies.

> everything should be a library.

That's exactly npm's problem, though. What everybody is avoiding to say is that you need a concept of "trusted vendors". And, for the "OSS accelerates me" business crowd, that means paying for the stuff you use.

But who would want that when you're busy chasing "market fit".

> That's exactly npm's problem, though.

I don't think that's the problem with npm. The problem with npm is that no packages are signed, at all, so it ends up trivial for hackers to push new package versions, which they obviously shouldn't be able to do.

slf4j is probably more common now than standard Logger, and it was a good night for those who used Logback as implementation.

> Doing it the right way would create friction, developers might need to actually understand what the code is doing rather than pulling in random libraries.

Then let's add friction. Developers understanding code is what they should be doing.

CTOs understand the high cost of ransomware and disruption of service.

This is how our security lead is. "I've identified X as a vulnerability, recommended remediation is to remove it." "We literally can't." He pokes around finding obscure vulnerabilities and recommends removing business critical software, yet we don't have MFA, our servers and networking UIs are on the main VLAN accessable by anyone, we have no tools to patch third party software, and all of our root passwords are the same. We bring real security concerns to him like this, and they just get backlogged because his stupid tools he runs only detect software vulns. It's insanity.

>Some 19 year old hacked together a new front end framework last week, better use it in prod because why not.

The thing is, you don't have to be this undiscerning to end up with tons of packages.

Let's init a default next.js project. How many dependencies are there?

react, react-dom, next, typescript, @types/node, @types/react, @types/react-dom.

OK so 7... seems like a lot in some sense but its still missing many reasonable dependencies. Some sort of styling solution (tailwind, styled components, etc). Some sort of http client or graphql. And more. But lets just use the base dependencies as an example. Is 7 so bad? Maybe, maybe not, but you need to go deeper. How many packages are there?

55. What are they? I have no idea, go read the lock file I guess.

All of this while being pretty reasonable.

Since Shai-Hulud scanned maintainers' computers, if the signing key was stored there too (without a password), couldn't the attackers have published signed packages?

That is, how does signing prevent publishing of malware, exactly?

Package registries should step up. They are doing some stuff but still NPM could do more.

And yet, there are times where all I've had access to was the stdlib. I was damn glad for urllib2 at those times. It's worth it to have a batteries included stdlib, even if parts of it don't wind up being the most commonly used by the community.

The fact that there is a 'urllib2' implies that there's a 'urllib', which tells us something pretty important about the dangers of kitchen-sink standard libraries.

How did Shai-Hulud get access to maintainers' computers?

Related: Rust Dependencies Scare Me [1]

[1] https://vincents.dev/blog/rust-dependencies-scare-me/

But nothing prevents a language to have rich and OPTIONAL stdlib, so that devs can choose different solutions without linking bunch of junk they do not use.

Really, good stdlib still allows you to use better suited 3rd party libraries. Lack of good stdlib doesn't add anything.

This is the philosophy used by the Java world. Big parts of the standard library are plugin-based. For example, database access (JDBC), filesystem access (NIO), cryptography (JCA). The standard library defines the interfaces and sometimes provides a default implementation, but it can be extended or replaced.

It works well, but the downside of that approach is people complaining about how abstract things are.

> if the signing key was stored there too (without a password), couldn't the attackers have published signed packages?

Yeah, of course. Also if they hosted their private key for the signature on their public blog, anyone could use it for publishing.

But for the sake of the argument, why don't we assume people are correctly using the thing we're talking about?

In past comments I said that a quick win would be to lean on certificates; those can't easily be forged once a certificate is accepted.