A critique of package managers

I see this a lot with Rust where I will depend on one or two external crates for a simple application and then I am shocked to see dozens of dependencies being pulled in when I go to build. I actually think Cargo's support for feature gates and conditional compilation could in theory be a strong mitigation against this as crates can avoid pulling in dependencies unless you actually need a feature that relies on them, but in practice it doesn't seem to work that way as I often see these complaints about Rust.

I sympathise with the arguments but IMO laziness will always win out. If Rust didn't have Cargo to automate dependency hell, someone would create a third party script to fill the gap.

It is an organizational not a technical problem.

When I worked at Google every single dependency was strictly vendored (and not in the mostly useless way that Cargo vendors things). There was generally only one version of a dep in the mono repo, and if you wanted something.. you generally got to own maintaining it, and you had to make sure it worked for every "customer" -- the giant CI system made sure that you knew if an upgrade would break things. And you reached out to stakeholders to manage the process. Giant trains of dependencies were not a thing. You can do that when you have seemingly infinite budget.

But technology can indeed make it worse. I love Rust, but I'm not a fan of the loose approach in Cargo and esp Crates.io, which seems to have pulled inspiration from NPM -- which I think is more of a negative than positive example. It's way too easy to make a mess. Crates.io is largely unmoderated, and its namespace is full of abandoned or lightly maintained projects.

It's quite easy to get away with a maze of giant transitive deps w/ Cargo because Rust by default links statically, so you don't usually end up in DLL hell. But just doing cargo tree on the average large Rust project is a little depressing -- to see how many separate versions of random number generators, SHA256, MD5, etc libs you end up with in a single linkage. It may not be the case that every single one is contributing to your binary size... but it's also kind of hard to know.

Understanding the blast radius of potential issues that come from unmoderated 3rd-party deps is I think something that many engineers have to learn the hard way. When they deal with a security vulnerability, or a fundamental incompatibility issue, or have to deal with build time and binary size explosions.

I wish there was a far more mature approach to this in our industry. The trend seems to be going in the opposite direction.

Rust’s big issue here is the anemic standard library. I think overall the strategy makes some amount of sense; since there’s so much crazy alchemy like depending on nightly, no_std, etc in Rust, including stuff in std has more downside in Rust than in a language that’s more stable like Go.

But it’s annoying to have to deal with 3 different time libraries and 3 different error creation libraries and 2 regex libraries somehow in my dependency tree. Plus many packages named stuff like “anyhow” or “nom” or other nonsense words where you need to google for a while to figure out what a package is supposed to do. Makes auditing more difficult than if your library is named structured-errors or parser-combinator.

I don’t like go programming language but I do like go tooling & go ecosystem. I wish there was a Rust with Go Principles. Swift is kinda in the right ballpark, packages are typically named stuff that makes sense and Swift is closer to Rust perf and Rust safety than Go perf and Go safety. But Swift is a tiny ecosystem outside of stuff that depends on the Apple proprietary universe, and the actual APIs in packages can be very magical/clever. ¯\_(ツ)_/¯

> If Rust didn't have Cargo to automate dependency hell, someone would create a third party script to fill the gap.

Possibly but not guaranteed. Some other languages without a built in package manager haven't had an external one manage to take over the ecosystem, most (in)famously C and C++, while others have.

I agree, though also I note Python has an extensive standard library and isn't much better in terms of package sprawl.

Yeah, Python is a cautionary tale here, and I think one that informed the Rust stance.

Python is much older than Go, and has had more packages move from 3rd party into the stdlib to become a "battery", and then atrophy over the years while people move back to 3rd party alternatives with more features that are actually receiving maintenance. Eventually some of those modules were removed from core.

Perhaps the Go model only works when you have a very dedicated core group (for Go, mostly Google employees) around to continuously build and maintain the Cathedral of the standard library + toolchain together. Golang feels very much like UNIX (eg FreeBSD) for this reason, and Rust/Python more like Linux.

Most language users will follow the "spirit" of the language - e.g. Bill is against package managers, people who use his language mostly agree with his ideas, and there's not a huge standard Odin package manager.

I rather appreciate that C and C++ don't have a default package manager that took over - yes, integrating libraries is a bit more difficult, but we also have a lot of small, self-contained libraries that just "do the thing" without pulling in a library that does colored text for logging, which pulls in tokio, which pulls in mio, which pulls in wasi, which pulls in serde, which is insane.

C and C++ do have package managers. It's just that these languages evolved for OS implementation and also that these package managers are old and stable so they have support for a lot of languages, so that you probably know them as OS package managers.

Which is the correct thing to do as things written in language X can depend on things written in language Y.

In many ways traditional Linux distros operate on similar model as I imagine googles monorepo. Both aim to this "globally consistent" dependency situation where you have one version of each library and you patch up things from upstream when they don't fit.

I feel we need more of these kinds of distros so you don't need to manage dependencies directly from upstream and deal with the integration effort yourself. What if we had a Rust disto following this same model, where there is only one version of each dep, some reasonable curation, and also you had nice clear release cycles? I feel that could real boon for the industry.

Unfortunately I think it'd be too much putting the toothpaste back in the tube at this point. The way people are used to working in Rust is like filling their shopping cart with crate treats and then gluing them together.

I dunno maybe what is needed is a crates.io alternative that is highly highly moderated and highly highly selective. A subscription service with a paid staff that manages the packages and makes sure their deps are minimal, consistent with each other, secure, etc.

I can see that being a service that some corporations might pay for. I just came off a gig at a medical devices company that was using Rust and the software BoM side of things kept me up at night. The list of dependencies in the root workspace was long, and in my imagination, full of terrors.

Maybe they'll be fine, but it's not a practice I would recommend if I were starting such a project from scratch.

The very sparse std is one of the few genuine mistakes I think Rust has made. I know the arguments for it, but I don't find them persuasive. A batteries included standard library, in my view, is just plain better and every modern language should have one.

The package manager for C/C++ is apt, or rpm, or whatever package manager your system uses. These package managers were designed for the world of C/C++ software so it's less surprising that these languages haven't found as much of a push towards language package managers.

It is not. Most distro have their own build scripts that target the specific library name that is needed. Dependencies is mostly done through a mix of convention, helper programs/scripts and compiler args.

Oh man I feel the opposite. Rust is an example where a sparse stdlib is clearly superior and better and more successful.

I mostly write C++ whose committee is incompetent and sniffs glue. And I deal a lot with Khronos committed who design pure garbage. “Design by Committee” is a pejorative for a reason.

Most distros have their own package manager including source packages and an automated way to go from there to binaries. So the build scripts you mention are part of the package manager.

Yes, but that's quite different from how language package managers are used. A typical C/C++ project doesn't tell you to build it via apt/rpm instead you are expected to provide the dependencies using whatever means you choose or they are included in the repo. This means you don't need to fight a package manager to get fully offline or reproducible builds for example.

RPM and APT packages are also usually not maintained by the upstream developer but by distro developers who care about making different packages work together so you don't get the dependency hell problem as a user.

Just as the Rust community has largely converged on tokio as the standard async runtime, is there any reason why there couldn't exist a community-developed "batteries-included" standard library other than writing a standard library being a tedious and thankless task?

But what batteries to include? I'm sure your list of batteries is probably very different from mine, reflecting the difference in the work we both do. A lot of times when I hear a language has "batteries included", it's a bunch of features for web devs and nothing I would consider a "battery" for my projects.

What do you consider your set of "Batteries Include"?

Basically the stuff in here: https://www.ros.org

Yes, C programs go to lengths to give the user choice. But it is recommended to use source packages for e.g. dpkg. And it's not too rare to have the .debian directory in the original VCS.