Buck2: Our open source build system

You cannot magick away complexity. Large systems (think thousands of teams with hundreds of commits per minute) require a way to express complexity. When all is said and done, you'll have a turing-complete build system anyway - so why not go with something readable

I seriously doubt there's a single repo on the planet that averages hundreds of commits per minute. That's completely unmanageable for any number of reasons.

It wouldn't surprise me at all if some large repos at Google or Facebook now get to that many, it's easy to do once you have robots committing code (usually configuration changes).

I didn't mean on average, but the build tool has to handle the worst case and I probably am understating the worst case.

I'd bet there are a more than a few repos that do get (at least) hundreds of commits as a highwater mark. My guess is lots of engineers + mono-repo + looming code-freeze deadline can do that like clockwork.

Edit: Robots too as sibling pointed out. A single human action may result in dozens of bot-generated commits

According to [1], in 2015 Google averaged 25 commits per minute (250000/7/24/60). I can imagine hundreds per minute during Pacific working hours today.

[1] https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...

IMO there's almost never a good reason to have automated commits in repos outside of two cases:

1) Automated refactoring

2) Automated merges when CI passes

Configs that can be generated should just be generated by the build.

But that's a different topic

In the case of a monorepo, that's exactly the case why the build system shouldn't be overly complex. If you're expecting random people to make changes to your stuff, you shouldn't be burdening them with more complexity than necessary.

The monorepo case is also a little bit outside what I was originally talking about. I was mostly refering to individual services/libraries/apps

There are at least two other hugely important use cases you missed:

- automatic security / vendoring updates (e.g. https://github.com/renovatebot/renovate)

- automated cross-repo syncs, e.g. Google has processes and tools that bidirectionally sync pieces of Google3 with GitHub repos

On the contrary, large systems have to restrict what their build system does because otherwise the complexity becomes unmanageable. I used to work on a large codebase (~500 committers, ~10MLOC) that had made the decision to use Gradle because they thought they needed it, but then had to add increasingly strict linters/code review/etc. to the gradle build definitions to keep the build maintainable. In the end they had a build that was de facto just as restricted as something like Maven, and the Turing completeness of Gradle did nothing but complicate the build and slow it down.

And sure, maybe having a restricted build definition (whether by using a restricted tool or by doing code review etc.) moves the complexity somewhere else, like into the actual code implementation. But it's easier to manage there. The build system is the wrong place for business logic, because it's not somewhere most programmers ever think to look for it.

No no no no. The more code you have, the more you have to constrain the builds.

I understand where the sentiment comes from, having seen one too many example of people struggling to implement basic logic in cmake or groovy, that would be a oneliner in python. But completely opening up the floodgates is not the right solution.

Escape hatches into GP languages can still exist but the interfaces to them need to be strict, and it’s better people see this boundary clearly, rather than limping around trying to do GP inside cmake and failing on correctness anyway. Everything else should like parent say just be a list of files.

Dependencies need to be declarative and operations hermetic.

Otherwise the spaghetti of complexity will just keep growing. Builds and tests will take forever due to no way of detecting what change affects which subsystem, what can be parallelized and even worse when incremental builds stop working.

By constraining what can be done, you also empower developers to do whatever they want, within said boundaries, without having to go through an expert build-team. Think about containers, it allowed every team to ship whatever they want without consulting the ops team.

> The more code you have, the more you have to constrain the builds.

That works if you have one team - of if all teams work the same way. If you have multiple teams with conflicting requirements[1], you absolutely should not constrain the build because you'd be getting in the way.

1. E.g. Team A uses an internal C++ lib an online service and prefers an evergreen version of it to be automatically applied with minimal human involvement. Team B team uses the same lib on physical devices shipped to consumers/customers. Updates are infrequent (annual), but have to be tested thoroughly for qualification. Now your build system has to support evergreen dependencies and versioned ones. If you drop support for either, you'll be blocking one team or the other from doing work.

> IMO there's almost never a good reason to have automated commit

This depends entirely on the quality of dev tools available.

Also, commit =/= shipped code: you may have a automated commits and keep a human in the loop before shipping, by way of rejectable Pull-Request (or the proprietary equivalent).

A simple library upgrade will result in a wave of commits/bot-authored PRs

1. Human makes a change to a core library, changing it from v1 to v2

2. Bot identifies all call-sites and refactors to v2-equivalent, creating 50 PRs for 50 different teams.

One change, 51 commits.