Buck2: Our open source build system

> Build systems stand between a programmer and running their code, so anything we can do to make the experience quicker or more productive directly impacts how effective a developer can be.

How about doing away with the build system entirely? Build systems seem like something that shouldn't exist. When I create a new C# .NET app in Visual Studio 2019, what "build system" does it use? You might have an academic answer, but that's beside the point. It doesn't matter. It just builds. Optimizing a build system feels like a lack of vision, and getting stuck in a local maxima where you think you're being more productive, but you're not seeing the bigger picture of what could be possible.

How do you think VS builds your code?

Visual Studio is a build system. And about eleven other things.

I guess you could write machine code by hand if you don't want to build it. Otherwise, what specifically do you propose to do away with build systems?

How does Visual Studio convert a solution to an executible?

If you look around the toolbench, and you can't figure out who the build system is ...

I am not sure I understand your point. I click build in VS, whatever VS does it takes 2 minutes.

Supposedly the C# compiler can compile millions of lines per second, but my not millions of lines of code project takes a minute to compile so it must be wasting time doing something, and could use some optimization.

Exactly right, people are very confident that they need a non-abstracted build system, and that it needs to be replaced every N years. I'm calling that into question, and people are confidently telling me I'm wrong, without the ability to explain why. So your link is very appropriate.

The point is it doesn't matter. What it's doing internally isn't interesting to me, a programmer. I'd rather not think about it. Unfortunately, people have decided that microoptimizations like writing declarative build definitions by hand is beneficial, so I've had to learn many build systems.

Using a compiler. Less snarkily: It uses a compiler, and a black box which I don't have to think about or configure. There is technically a build system in there, but again, I don't have to think about it or configure it, unless I'm doing something really weird (and in that case, Visual Studio has GUI options for that).

Why should I care what the build system is? Please use your words.

Okay you got me, I don't want to get rid of build systems. I want to stop having to think about them, because it's a waste of my time.

Regrettably, your observation doesn't prove your point. It's common knowledge that Visual Studio is very slow. It's entirely unknown how much of that slowness is related to the build system (using that term in the same sense as buck2).

To some, the bigger picture of what could be possible includes using multiple languages, multiple build steps, etc.

Refining individual ecosystems to be effortless to use is one thing, and it’s a goal worth pursuing for the benefit of those who are happy in that ecosystem. For those who rely on bringing different technologies together, though, there can be a lot of complexity and nuance.

The point of these projects is to take that complexity and nuance and provide an effective “super ecosystem” to piece them together in. That’s why some of the biggest proponents of these technologies are Google and Meta - they have many interacting parts in many different languages that need to come together and work effectively.

Have you ever worked on a project that has to combine code from different languages into one cohesive application or tool? Have you ever had to build a binary that needs to end up in an installer package with some custom install scripts and also needs to support multiple end user OSs?

It's nice that you've lived in a world where you haven't had to concern yourself with the concerns of how your code is built, but please understand that some of us actually need or want to delve into this. It can't always be magic.

Building different languages into one cohesive application or tool is a procedural process. I build the code in language A, I build the code in language B, I package them together in some way, and do something with the result. I don't need a build tool with a custom declarative language for that, I can just use a BASH script. The problem is people want to do both high-level merging such as this and low-level compilation management using the same declarative language. It should really be two things.

So what you're saying is this is unfit for compiling projects, but could be used in a FAANG-scale environment to allow developers to aggregate lower-level efforts in a declarative way? I like that idea.

Sure:

Whatever code we write as developers is only very tip of a the iceberg of software that comprises any substantial application. Controlling what goes into that iceberg, and how it's assembled is an essential part of the engineering of software. The details and quality of your build system determine the composition and construction of that iceberg, not to mentioned the reliability and velocity of your development process.

Even 'basic' local build systems like CMake, maven/gradle/ivy, sbt, lein, cargo, go, ... bridge dependency management and task execution. They decide what goes into the software artifact you ultimately distribute (or deploy), and how that's assembled.

At the scale of buck, bazel, ... tools of that shape are necessary to make forward progress in that's composed of internal dependencies that are managed by different teams, written in different languages, targeting a variety of environments, that are so numerous they require distribution to complete in reasonable timeframes, and require absolute.

I'm not VS/C# user, but MSBuild is definitely a build system, and both in and the developer definitely have care about these complexities, even if they come under the heading of "IDE" instead of "Build System".

Also:

As the joke in my first comment implied, if you can't identify the build system, you're probably the build system

I think across this thread you're thinking about the run-a-graph-of-tasks aspect of build systems. I'd agree with you that's mostly-uninteresting. However, that's also the least significant aspect of what these build systems do. Most of the work is _determining_ the graph tasks that need to be run, and therefore a lot of the differentiation is in the ability to write new rules to guide that process and in the engine driving it.

The way it builds your software is also a declarative build system: MSBuild. It happens to be configured in XML instead of a Python-like language, and the IDE integration is probably better than what you get from Buck or similar.

Building software can be complex sometimes, and _pretending_ it's a "microoptimization" to care about that is unnecessarily diminutive.

I think this is a case of YAGNI. Yes, there are cases where you have a big graph of sources and products and you can't plan up-front what you'll need. I.E. you're at a megacorp and your dependencies are created by some branch of the organization that is actively hostile to yours. I dunno. But I think that most people just don't have those problems.

I'm starting to understand. This is a problem at FAANGs, and I simply haven't worked at a FAANG.

Well first, there is a build system in your VS. The fact that you don't know about it does not magically make it disappear. Some people have to understand some of it. For instance the person who maintains your project for you, apparently (because you'd rather not think about it).

It's like if I said "I don't really know to understand the details of how a CPU works in my daily job. What about we just removed CPUs entirely?".

For the same reason someone creates a new programming language every second day: some people find it interesting, some people like to learn new languages, and sometimes a new language actually brings something really valuable.

Is it worth putting those resources into a new build system? Apparently Facebook thinks so. You don't have to agree, you can use Visual Studio.

It just seems like you say "if I don't care about build systems, nobody should".

> When I create a new C# .NET app in Visual Studio 2019, what "build system" does it use?

...msbuild?

> I package them together in some way

That's a build system!

> I can just use a BASH script

Yep you can. Many build systems start out like that. Complexity tends to creep in.

(Note: I work at Meta, on the buck team)

This isn't really just a FAANG thing. I've seen this throughout my career before the FAANGs at much smaller places, but projects needing multi-platform/multi-language/multi-arch scenarios.

You do have a point, that the issue is that one shouldn't have to learn the depths of a build system to do what one needs. The trade off is how are the knobs exposed to a developer.

Another way to think about it, these domain specific languages like Starlark and otherwise are just where the knobs are being stored. As much as Visual Studio ends up writing these to vsproj/vcproj or msbuild files, it's just that these are stored in another form.

Software scales up in the size of the iceberg of your deployment, and out in the number of teams working on it. Buck (and bazel and pants) solve for scaling both directions at once.

Whether you should care about the details and knobs of your build system is strictly a function of the first kind of scaling. Every team building software of appreciable size should think a bit about their build system, and even small teams might benefit from Buck or friends if their iceberg is big enough.

Visual studio is your build system. What if I want to use, develop, or depend on a project that wasn't built with visual studios preferences in mind?

I mean I don't know how to use visual studio to build software, but I get along fine, so why should we optimize for your preferred interface and not another one that might be better?

A shell script that copies files into a directory may technically be a "build system" in the same way that a hot dog it technically a sandwich. Obviously (question mark?) I'm not talking about that.

To clarify, I don't think that complex build systems are only used at FAANGs, I think that their usefulness is only realized at FAANGs. From what I've seen at smaller companies, most people are using these tools because everyone else does, not because their software production suffers from the kinds of problems that these tools solve. These tools also seem to provide the mental framework that people need to think about their software production processes, so I guess there's that.

For C++, the #include's kind-of-sort-of tell the story of what needs to be included in the build, and morally, it seems wrong that you should have to repeat that information anywhere. I recently watched a talk from cppcon from the people at tipi.build, which seemed to go along these lines. They've tried to completely tear away the entire notion that you should be writing separated build files that re-describe what you are already telling the compiler with your #include's.

Well, at the top-level of this comment section I see people cheering this on, and people excited to try this out in their own. I don't think they all work at Facebook, so maybe they should be closer to me than Facebook on this matter.