Wild – A fast linker for Linux

Ever since mold relicensed from AGPL to MIT (as part of mold 2.0 release), the worldwide need for making another fast linker has been greatly reduced, so I wasn't expecting a project like this to appear. And definitely wasn't expecting it to already be 2x faster than mold in some cases. Will keep an eye on this project to see how it evolves, best of luck to the author.

Maybe I'm holding it wrong, but mold isn't faster at all if you're using LTO, which you probably should be.

I think we're talking about non-release builds here. In those, you don't want to use LTO, you just want to get that binary as fast as possible.

Note that Mold has no interest in becoming incremental, so there is a big reason there for another linker to exist. I find it kind of embarrassing that MS' linker has been incremental by default for decades, yet there's no production ready incremental linker on Linux yet.

Mold will be faster than LLD even using LTO, but all of its benefits will be absolutely swamped by the LTO process, which is, more or less, recompiling the entire program from high-level LLVM-IR. That's extremely expensive and dwarfs any linking advantages.

So the benefit will be barely noticable. As another comment points out, LTO should only be used when you need a binary optimized to within an inch of its life, such as a release copy, or a copy for performance testing.

Yeah, if you're development process requires LTO you may be holding it wrong....

Specifically, if LTO is so important that you need to be using it during development, you likely have a very exceptional case, or you have some big architectural issues that are causing much larger performance regressions then they should be.

I'm pretty sure that's a typo, and "incremental" was meant to be included in that sentence.

Being able to choose a middle ground between development/debug builds and production builds is becoming increasingly important. This is especially true when developing in the browser, when often something appears to be slow in development mode but is fine in production mode.

WebAssembly and lightweight MicroVMs are enabling FaaS with real time code generation but the build toolchain makes it less appealing, when you don't want it to take half a minute to build or to be slow.

Yes, indeed I accidentally forgot about incremental there.

Why so hostile? Have a break, go look at the clouds, they are beautiful today!

Yes, I missed a word. And I believe pretty much everybody else realized what I meant to say.

Feel free to point me in the direction of a production grade incremental compiler that can run on Linux, GNU or otherwise.

Agreed. Both fast and small are desirable for sandboxed (least authority) isomorphic (client and server) microservices with WebAssembly & related tech.

Wait a minute, it’s possible to relicense something from GPL to MIT?

Yes. Generally you need permissions from contributors (either asking them directly or requiring a contribution agreement that assigns copyright for contributions to either the author or the org hosting the project), but you can relicense from any license to any other license.

That doesn't extinguish the prior versions under the prior license, but it does allow a project to change its license.

"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."

https://news.ycombinator.com/newsguidelines.html

If you're debugging, and your bug only reproduces with LTO enabled, you don't have much of a choice...

Additionally the way precompiled headers are handled in Visual C++ and C++ Builder have always been much better than traditional UNIX compilers, and now we have modules as well.

It has to be a candidate for the longest biggest gap in build tooling ever.

Username checks out.

And factual.

Sure, for that 1% of the time.

Yes if you are the only developper and never received nor accepted external contributions or if you managed to get permission from every single person who contributed or replaced their code with your own.

> or if you managed to get permission from every single person who contributed

This makes it sound more difficult than it actually is (logistically); it's not uncommon for major projects to require contributors to sign a CLA before accepting PRs.

That depends on how old and big is the project. For example Linux is "stuck" on GPL2 and even if they wanted to move to something else it wouldn't be feasible to get permission from all the people involved. Some contributors passed away making it even more difficult.

...which takes these remaining 99% of a development time...

Not exactly “stuck” since they very explicitly do not want to move to GPL 3.

OTOH even lld, fast but fairly slower than mold, is already incredibly faster than MS's linker even without the incrmeentality. Like, I'm routinely linking hundreds of megabytes in less than a second anyways, not sure incrementality is that much worth it

Not a rhetorical question: Could it be that part of the speed difference is due to the file system speed? I was shocked when I saw how much modern(ish) Windows file systems were slower than modern(ish) Linux ones.

Why does AGPL Vs MIT matter for a linker?

it's usually windows defender and virus scanning that causes these massive slowdowns.

Corps don't want to have to release the source code for their internal forks. They could also potentially be sued for everything they link using it because the linked binaries could be "derivative works" according to a judge who doesn't know anything.

I think you should get new lawyers if this is their understanding of how software licenses work.

I'm waiting for 'linker-guy to weigh in, personally.

Surely your LTO bugs are not so easy to fix that they take less time to resolve than linking itself does.

what is the status of Windows support in mold? reading the github issues leads to a circular confusion, the author first planned to support it, then moved Windows support to the sold linker, but then sold got archived recently so in the end there is no Windows support or did I just misunderstand the events?

That’s exactly the problem “Dev Drive” is intended to solve I believe. I haven’t tried it myself.

https://learn.microsoft.com/en-us/windows/dev-drive/

See for example https://opensource.google/documentation/reference/using/agpl...

> Code licensed under the GNU Affero General Public License (AGPL) MUST NOT be used at Google.

Corps want to be able to release and use tools that take away the freedoms that GPL-family licenses provide. Often this results in duplication of effort.

This is not theoretical; it happens quite frequently. For toolchains, in particular I'm aware of how Apple (not that they're unique in this) has "blah blah open source" downloads, but often they do not actually correspond with the binaries. And not just "not fully reproducible but close" but "entirely new and incompatible features".

The ARM64 saga is a notable example, which went on for at least six months (at least Sept 2013 to March 2014). XCode 5 shipped with a closed-source compiler only for all that time.

It’s their loss

So they donate money instead of code? The project somehow benefits from the switch to MIT?

They don't have to release source for internal forks.

And as a result of this, if you want to optimize your program to be fast on Windows, you will make very different optimization decisions than on other platforms.

For example, Windows runs virus scans on close(), which makes it very slow. This means that sometimes it makes sense to have one or more background threads exclusively dedicated to closing files.

There's a good talk on this at https://www.youtube.com/watch?v=qbKGw8MQ0i8

Is it? Because open source tools re-licensing themselves to be more permissive would seem to indicate whose loss it really is.

Hmm, my naive summary of AGPL is "If you run AGPL code in your web backend you are obliged to offer the backend source to everyone using a web client". No wonder it's explicitly forbidden at Google.

What does that mean for a linker? If you ship a binary linked with an AGPL linker you need to offer the source of the linker? Or of the program being linked?

In practice I think it's pretty much equivalent to the GPL for a linker. But I can understand why people in commercial settings are wary of this license.

Embrace, extend, extinguish. it could take about a century, but every software company (hardware maybe next century) is in the process of being swallowed by free software. Thats not to say people can’t carve out a niche and have balling corporate retreats for a while.. until the sleeping giant wakes up and rolls over you.

The way precompiled headers work in C++ is a bit of an ugly hack. And worse, it's almost as slow as just compiling them all every time anyway.

Only in traditional UNIX, which I mentioned.

Borland and Microsoft compilers have been dealing with them just fine since at very least 1994, when I started using Turbo C++ 3.1 for Windows.

Even if they wanted to move to another license (which they don't), they wouldn't be able to do. So sounds exactly like they're "stuck", regardless of what they want.

might have been a problem in Bun that was fixed by: https://github.com/oven-sh/bun/pull/16747

They do if they're AGPL licensed and the internal form software is used to provide a user facing service.

iirc the mold author wanted to make money off of it (and I dont blame him).

AGPL is avoided like the plague by big corps: same big corps are known for having money to pay for licenses and sometimes (yes, I look at you Amazon) being good at deriving value from FLOSS without giving back.

iirc AGPL was used so everyone can just use it, big biz is still compelled to buy a license. this has been done before and can be seen as one of the strategies to make money off FLOSS.

It helps. Install your games on it, too.

NTFS and ReFS are high performance file systems. But yes, it is due to file system filters, which OOTB means Windows Defender, though that can be extended by 3rd parties, including Sysinternals utils such as procmon.

Microsoft built a very extensible I/O stack and prior to Defender/prior to SSDs it really wasn't very noticable... back when it was originally designed well through the 90s and early 00s.

Unfortunately it is now noticable despite being an otherwise smart design. Which means Windows and/or NTFS are blamed as being slow, neither of which has any basis in fact when we look at the overall design of Windows' subsystems/VMM in comparison to macOS/Linux.

It sucks. You've got great plumbing in Windows with a shit shell on top.

You've just made a pretty outrageous claim without evidence that would require a lot of effort on my part to refute, so I'll just go with: if you say so.

Under what circumstances would commercial companies be required to buy a license?! If they provide Linking as a Service?

> you're development process requires LTO you may be holding it wrong....

Not necessarily. LTO does a very good job of dead code elimination which is sometimes necessary to fit code in microcontroller memory.

They probably wont NEED a license, but --as said-- big corps dont touch AGPL with a ten foot pole because legal. So it's just to shut up legal, most likely.

This might indicate moreso that they believe they won't lose anything by the transition and users might ultimately benefit

How is the problem of "you signed a CLA without authorization by your employer to do so" solved? I'm mostly asking because I saw the following:

"I will not expose people taping out Hazard3 to the possibility of your employer chasing you for your contribution by harassing them legally. A contribution agreement does not solve this, because you may sign it without the legal capability to do so (...)"

https://github.com/Wren6991/Hazard3/blob/stable/Contributing... (this is I believe the repo with design for riscv cores running on RPi Pico 2)

These are the ones I refuse to contribute to.

> Yeah, if you're development process requires LTO you may be holding it wrong....

I spent a few months doing performance optimisation work. We wanted to see how much performance we could wring out of an algorithm & associated data structures. Each day I’d try and brainstorm new optimisations, implement them, and then A/B test the change to see how it actually affected performance. To get reliable tests, all benchmarks were run in release mode (with all optimisations - including LTO - turned on).

Disabling Defender on directories involved in compiling and linking speeds up compile and link times by a factor of 5x or 6x. It's not a subtle difference.

I agonize over the insanity of removing Defender protection on precisely the files that are most vulnerable on my computer each time I do it. But I do it anyway.

Interestingly, Android Studio offers to turn off Defender protection for development directories for you, and does so each time you load a project, if you haven't already done so. So I no longer feel like I'm alone in my insanity.

I'm wondering if you've ever actually asked a real corporate lawyer for an opinion on anything relating to GPL licenses. The results are pretty consistent. I've made the trip on three occasions, and the response each time was: "this was not drafted by a lawyer, it's virtually ininterpretable, and it is wildly unpredictable what the consequences of using this software are."

Why do some companies engage with it then?

But then it isn’t “internal”…

Eh, all the GNU family of licenses were drafted by lawyers.

Just using any Copyleft software has no legal consequences (copyleft licenses kick in when distributing, not using them).

Instead of spreading FUD you could go read the AGPL.

You should be using LTO where incremental build times are a concern, i.e. for development builds.

And for realease builds link time is hardly a concern.

Free software basically only exists because it’s subsidized by nonfree software. It also has no original ideas. Every piece of good free software is just a copy of something proprietary or some internal tool.

It’s too hard to determine what pieces of your stack interact with public-facing services, particularly in a monorepo with thousands of developers. The effort involved and the legal risk if you get it wrong makes it an easy nope. Just ban AGPL.

The effort involved, and legal risk is exactly the same as for any Copyleft license. If you don't know what your stack is doing, that is the problem -- not the license.

Do you have any kind of source for such claims?