Zig is hard but worth it

I've lately thought that a package manager is as essential to a new language as a standard library. I would also add a LSP and standard code formatter to that list.

It is a bit unfortunate because all of the above is a pretty tall order. We're getting to the point that new languages are expected to boil the ocean by the time they reach 1.0

Zig has a pretty decent LSP.

I'm not so sure a package manager is really all that essential; it can certainly be convenient but especially in the space Zig is looking at it's pretty workable without one (without complex deep dependency trees you can use git submodules or just copy a directory). Or let me put it this way: I never really missed a package manager in Zig.

> I'm not so sure a package manager is really all that essential;

For open source software (libraries or programs, that depend on other libraries or programs), it is essential (if you're not distributing single functions like with Unison). For closed source it doesn't matter that much.

package managers are relatively easy to put together and release with your language—provided that you want your new package manager and surrounding ecosystem to work exactly the same as other, popular ones, without putting any effort into improving the status quo. doing better than other languages' package managers takes significant effort, because it's both a computer engineering problem and a social engineering problem.

it's nice when a new language has a package manager right out of the gate, but I would like to see more new languages take a more measured approach and aim to significantly improve upon past efforts, instead of merely replicating them out of some sense of obligation.

"cp -r ~/some-lib ~/my-project/" works well enough if some-lib doesn't have dependencies on its own. Or git submodules if you want something a bit more fancy. I sometimes do this even for languages with package managers, as it avoids a world of complexity.

Obviously a package manager is useful, but Zig is relatively low-level and long dependency chains are much less common than in e.g. Python, Ruby, and of course NodeJS. So I'd argue it's not essential. All the other things mentioned in the to top comment are far bigger issues IMO.

> I've lately thought that a package manager is as essential to a new language as a standard library. I would also add a LSP and standard code formatter to that list.

Agreed. Especially on a formatter. The number of code review comments it cuts out is incredibly time and energy saving.

> It is a bit unfortunate because all of the above is a pretty tall order. We're getting to the point that new languages are expected to boil the ocean by the time they reach 1.0

In order to get into production? yes. there are minimum requirements that must be met. those are higher now than in the past, because of lessons learned the hard way. those problems have been solved (for some value of solved) in existing ecosystems -- a new language wont change the need for them to be solved.

it shatters the dream of hacking up something on a weekend and having it turn into a hit success, but it also removes the case of hacking up something in 10 days and suffering the consequences for the next 30 years.

Until they have what you mentioned, the languages aren't ready for large scale use -- they need to grow into it. They can be useful prior to that -- enthusiasts and early adopters can reap what benefits are provided. That adoption is what fuels the development of things like a standard code formatter.

edit: fixed omission of a unit of time after '30'

Why? A submodule can be frozen at a given commit. They're trivially updated or rolled-back, too.

I disagree. I actively avoid languages that rely on package managers simply because they only give the illusion of being beneficial. It ends up being more boilerplate I have to learn to use an ecosystem, because they don't actually solve dependency hell and now there's a whole additional complicated tool with its own DSL I have to contend with in order to fix what's broken (or even diagnose issues.)

The more peripheral crap I have to deal with to use your language, the less I'm likely to use it in the first place. I don't need, want or care to learn yet another idiosyncratic fragile system. Finding source tarballs is a complete non-issue, and inevitably I'm going to have to manually build things anyways to figure out what erroneous assumption a library is making about the underlying system which is causing the build to fail or the runtime to crash, so the package manager just ends up being extra steps. Without fail, that has always been my experience with language package managers.

In the pursuit of making things simpler, we're really just making them harder.

I'm not sure I understand this issue. The existence of a standard package manager doesn't prevent you from manually vendoring dependencies if you want to. I have personal experience working on node.js projects where several dependencies were just `cp` into a directory and treated like local modules, no npm involved at all.

It's like `apt` or `brew` for system dependency management. It is there if you want it but you can just as well download a tar and config/make/install yourself if you want.

In many ways, it is like a standard lib. No one forces you to use it. If you prefer the simplicity of your own implementations then I see no reason why you can't just write your own.

But when you want the advantages of a package manager, and there are advantages that you may not appreciate but others do, then having a standard one built into the language feels preferable to having a dozen non-standard independently competing variations that the community cobbles together.

The cynic in me would say that including a standard package manager is absolutely necessary to stop several of them popping up every year :)

Would love to understand better what exactly is language specific about package managers? You would think a winner would emerge in this space like one has in version control to be used with all languages, just separate repos for different ecosystems?

Just curious, not snide: what language fits this criteria?

C/C++? Using Python 3.7+ in a way that completely ignores Pip, because the stdlib is now quite expansive?

> but it also removes the case of hacking up something in 10 days and suffering the consequences for the next 30 years

Ha, I thought this sounded distressingly familiar!

"In September 1995, a Netscape programmer named Brandan Eich developed a new scripting language in just 10 days. It was originally named Mocha, but quickly became known as LiveScript and, later, JavaScript."

> new languages are expected to boil the ocean by the time they reach 1.0

The reason is that there are existing high-quality languages/ecosystems.

(Which is a good thing!)

Must be C/C++.

I agree entirely. Relying on a package manager in order for a language to be useful indicates to me that there's a deficiency in the language.

I was also wondering this. Why does every language need to reinvent the wheel?

Different languages do imports differently. There are different constructs for “exports” and modules and namespaces that prevent a common single directory structure.

Pip used to store packages in a global location, now most of python used a Virtual Environment per project.

Node uses a “vendor” directory within the project structure. This is probably the easiest case.

Go used to store everything globally in the “go path” but now with go modules it does something else.

Java doesn’t need source code, just single JAR files, but it needs it within the classpath.

C/++ is very flexible, but varied depending on how you build and set up your project.

Swift/ObjC requires doing whatever apple wants and dealing with their tooling.

Everything is different. If you want “one winner” the closest you get it is the system package manager (of which multiple exist) and pointing your project to its package cache. But not all system package managers handle multiple concurrent versions of the same package.

Maybe one day people will standardize against Nix which seems to be the closest?

Python is quite often being used without pip, as the modules are often packaged by distros already (and in many distros they have to be packaged this way in order to include the application that uses them in the repos).

The problem with source tarballs is that you need to then manually watch out for updates and bugfixes for every dependency you pull in. You also need to deal with the bespoke build system of your packages of choice or manually build something compatible with your project. You can also choose not to and skip vulnerable, buggy code for a few years like some companies do, but that's hardly an advantage. You also need to ship (and possibly license) all of your dependencies.

Alternatively, you can let the system package manager do all the hard work for you. That works great, as long as you only target one OS or put in the work to maintain compatibility with multiple OS releases.

My experience with languages without package managers is that large, maintained projects all invent their own package manager (remote git repos? shell scripts downloading files? a two-stage build process written in the language itself?) in combination with some sort of Makefile generating hell script that breaks for a while when a new OS release comes out.

This approach works if you're the entire system. SerenityOS can do this, because it's basically an entire operating system and userland all inside one repository. ChromeOS can probably do it as well, since the app portion isn't running OS-native code anyway. For everyone else, it's just making someone else do the work of the package manager manually.

> I've lately thought that a package manager is as essential to a new language as a standard library.

This was the approach Ceylon took. Sadly, despite a lot of effort, the language never took off.

Odin

Is every language supposed to come with HTTP and TLS stack, clients for every database, de/serializers for every format, every image and video codec, every de/compressor, GUI toolkits, 3D rendering, Bluetooth… where do you stop?

And then how do you maintain all of this bloat to a competitive level, so that users don't need to reach for faster/newer alternative packages anyway?

And how do you maintain portability and backward compatibility of a language with such a vast API surface?

In modern development there's just too much stuff to keep everyone happy with batteries included. Sooner or later users will need something that isn't built in, and that will cause pain and crappy workarounds if the language doesn't have first-class support for dependencies.

I suppose that I just never had a problem maintaining any dependencies in code I write. Package managers have long been a bit of a pain for me as a developer, and with some languages (like Python), they are a huge PITA for me as a normal user of applications.

So overall I don't view them in a very positive light. They're something I have to put up with.

No matter, it is what it is. Carry on. :)

> I'm not so sure a package manager is really all that essential

I agree with you, but this is subjective. Not having a package manager will probably turn off many from the language. But it’s OK for the Zig folks to make a call that a lot of people won’t agree with if it doesn’t fit their vision of the language.

Not sure about lsp, but I think if you defined your language in tree sitter, you might be able to define a basic autoformatter generically on tree sitter to accelerate bootstrapping your language.

You could also use an existing language agnostic package manager like nix, guix, or conda to bootstrap your language package manager.

Lsp is something I don't know of a way to make that easy without overly constraining the design space.

You could manage it like .net did before chocolatey/winget -- pretty much a free-for-all, just add yourself to the COM list through whatever means you want and have at it

But that was a mess. I still have nightmares from dealing with windows assembly cache issues.

Honestly it reminds me of some of the shit I've had to deal with with pip (what do you mean you can't resolve this dependency???)

When Python was originally designed disk space was precious, and access to the Internet was rare, and its dependency management was designed for that world. Its multiple retrofitted package managers never fully fixed it.

However, better integrated package managers can work well. In case of Node.js and Cargo, the main argument against them is that it's too easy to add dependencies.

> Agreed. Especially on a formatter. The number of code review comments it cuts out is incredibly time and energy saving.

Are people realistically reviewing code formatting? As long as people aren't making egregious violations I generally don't care if someone leaves a brace on the same line or writes a one-line if statement. I tend to review the overall design and look for bugs and edge cases that might've been missed. If somebody told me they didn't like the way I formatted the code then they've got their own editor and are welcome to change it if they want to be petty.

Absolutely. Inconsistency is sloppy, increases cognitive burden while reading code, and can hide bugs.

Everyone should take the time to setup their editor to format their code consistent with the project style, it doesn't take long at all.

Nobody is reviewing it. They are rejecting malformatted code in CI step.

You're looking at it the wrong way

People review code. As soon as that happens, formatting is part of the conversation. Either consciously or subconsciously

Fundamentally the problem with python is that it's packages are global and every workaround except ~docker can't fix it except in a hacky way since the package resolution is specified in the language

A lot of what a package manager does isn't really language dependent. It seems like it would be possible to build a generalized package manager that has hooks for a language specific plugin to control how the package is extracted, how to resolve transitive dependencies, and how to do any additional build steps. It would be sort of analogous to how instead of building a full IDE, you can implement an LSP server and get support from existing IDEs and editors.

The point of the auto formatter is to get those slight differences out of the way so your Brian can focus on parsing and understanding the important parts

High level: I appreciate the point being made here. Challenge: If true, why does it not already exist? Idea: Apache Maven repositories use pom.xml to define dependency trees. Why don't these get used more for non-Java'ish languages? Mind you, I am not advocating to use Apache Maven as a build tool. I am noting that (remote) repos and their dependency metadata are a huge achievement in that ecosystem.

In addition to reviews, I've started to really enjoy format-on-save as a way to save keystrokes. Instead of manually formatting my code (even for myself), I can take the shortest editor path to the AST that I want and then have everything else snap into place

> If true, why does it not already exist?

Because no has put in the effort to build it and get it adopted. You could also ask why we didn't have something like LSP earlier. In order for such a project to become widely adopted it would need to work well for at least a few popular languages, and work at least approximately as well as the native package management solutions for those languages. And it would need to be easy to use. You can kind of use some existing tools, such as maven, bazel, even npm, for other languages, but it usually isn't as nice of an experience.

That's not the workflow your GP is describing.

Furthermore, having a package manager indicates there are some kind of specifications behind so each package and repository has an expected format (not necessarily _a_ standard package manager; in many ecosystems multiple managers share one standard). This helps vendoring a lot since the standard specification makes the vendoring process very easy to automate and validate. Dependency managers in their essence is really just automated vendoring (or vendoring is just less automated dependency management); it helps because you can pick it apart and choose to automate only the parts you want, even if you don’t like the fully automated solution.

I think JS was somehow salvaged after all that time. The event loop paradigm is lovely.

Makes me so sad that I can't just code up a parser and have a useable language. I have a lot of ideas, but heck, just syntax highlighting is... I don't know where to begin. And auto-completions. I need a tool that does 90% of that like PegJS or other parser-generators do for the parsing part. Don't see why a parser-generator can't also spit out an LSP. And if it knows what tokens are valid next, it should be able to generate completion suggestions. And if you have an AST, you can probably format the code half decently. And then I imagine it's possible to have a semi-universal package manager, I don't know why we need one per language, really. Aren't they all "put some code in some predefined location, and resolve these dependencies with semver please"? Maybe some extra junk for precompiled deps but still.

Tell us more, cause it’s not a problem I know

So you manually do what a package manager would do automagically and better. What do you think you are doing that is better than a program resolving dependencies, downloading h them and building them, putting them in the right location and tracking the versions used in a lock file

Try installing two pieces of software that depend on two different versions of tensorflow, then come back here.

I can install multiple python interpreters side by side each with independent packages and I can install different versions of the same python package for the same interpreter. I haven’t tried or had a need, by induction sounds possible :shrug:

Because LSP is not just "give me some semi-decent completions based on parsing tokens in this file". Proper LSP will treat code semantically, in case of e.g. Rust it may complete a method name from a trait method because you have the trait somewhere in scope, which comes from a third-party package which the LSP also parses, where it needs to expand some procedural macros to get to source code, etc etc.

We've been spoiled by good LSPs, perhaps. But again, it's not just "parse this file please".

Tensorflow wheels are bound to cuda installation which is driver/sudo level. It's a nightmare.