←back to thread

392 points mfiguiere | 2 comments | | HN request time: 0.001s | source
Show context
RcouF1uZ4gsC ◴[] No.35470953[source]
> Buck2 is an extensible and performant build system written in Rust

I really appreciate tooling that is written in Rust or Go that produce single binaries with minimal runtime dependencies.

Getting tooling written in for example Python to run reliably can be an exercise in frustration due to runtime environmental dependencies.

replies(3): >>35471099 #>>35471103 #>>35471569 #
rektide ◴[] No.35471103[source]
Personally it seems like a huge waste of memory to me. It's the electron of the backend. It's absolutely done for convenience & simplicity, with good cause after the pain we have endured. But every single binary bringing the whole universe of libraries with it offends.

Why have an OS at all if every program is just going to package everything it needs?

It feels like we cheapened out. Rather than get good & figure out how to manage things well, rather than driver harder, we're bunting the problem. It sucks & it's lo-fi & a huge waste of resources.

replies(7): >>35471298 #>>35471361 #>>35471433 #>>35471640 #>>35471685 #>>35472010 #>>35472197 #
bogwog ◴[] No.35471640[source]
I don't think that matters so much. For building a system, you definitely need dynamic linking, but end user apps being as self contained as possible is good for developers, users, and system maintainers (who don't have to worry about breaking apps). As long as it doesn't get out of hand, a few dozen MBs even is a small price to pay IMO for the compatibility benefits.

As a long time Linux desktop user, I appreciate any efforts to improve compatibility between distros. Since Linux isn't actually an operating system, successfully running software built for Ubuntu on a Fedora box, for example, is entirely based on luck.

replies(1): >>35471780 #
rektide ◴[] No.35471780[source]
There's also the issue that if a library has a vulnerability, you are now reliant on every static binary updating with the fix & releasing a new version.

Where-as with the conventional dynamic library world one would just update openssl or whomever & keep going. Or if someone wanted to shim in an alternate but compatible library, one could. I personally never saw the binary compatibility issue as very big, and generally felt like there was a while where folks were getting good at packaging apps for each OS, making extra repos, that we've lost. So it seems predominantly to me like downsides, that we sell ourselves on, based off of outsized/overrepresented fear & negativity.

replies(1): >>35472269 #
preseinger ◴[] No.35472269[source]
the optimization you describe here is not valuable enough to offset the value provided by statically linked applications

the computational model of a fleet of long-lived servers, which receive host/OS updates at one cadence, and serve applications that are deployed at a different cadence, is at this point a niche use case, basically anachronistic, and going away

applications are the things that matter, they provide the value, the OS and even shared libraries are really optimizations, details, that don't really make sense any more

the unit of maintenance is not a host, or a specific library, it's an application

vulnerabilities affect applications, if there is a vulnerability in some library that's used by a bunch of my applications then it's expected that i will need to re-deploy updated versions of those applications, this is not difficult, i am re-deploying updated versions of my applications all the time, because that is my deployment model

replies(2): >>35472452 #>>35475496 #
rektide ◴[] No.35472452[source]
Free software has a use beyond industrial software containers. I don't think most folks developing on Linux laptops agree with your narrow conception of software.

Beyond app delivery there's dozens of different utils folks rely on in their day to day. The new statically compiled world requiring each of these to be well maintained & promptly updated feels like an obvious regression.

replies(2): >>35472926 #>>35472959 #
howinteresting ◴[] No.35472926[source]
Again, there is no alternative. Dynamic linking is an artifact of an antiquated 70s-era programming language. It simply does not and cannot work with modern language features like monomorphization.

Linux distros are thankfully moving towards embracing static linking, rather than putting their heads in the sand and pretending that dynamic linking isn't on its last legs.

replies(1): >>35473824 #
PaulDavisThe1st ◴[] No.35473824[source]
Whoa, strong opinions.

Dynamic linking on *nix has nothing to do with 70s era programming languages.

Did you consider the possibility that the incompatibility between monomorphization (possibly the dumbest term in all of programming) and dynamic linking should perhaps saying something about monomorphization, instead?

replies(1): >>35474200 #
howinteresting ◴[] No.35474200[source]
> Dynamic linking on *nix has nothing to do with 70s era programming languages.

Given that dynamic linking as a concept came out of the C world, it has everything to do with them.

> Did you consider the possibility that the incompatibility between monomorphization (possibly the dumbest term in all of programming) and dynamic linking should perhaps saying something about monomorphization, instead?

Yes, I considered that possibility.

replies(1): >>35474351 #
PaulDavisThe1st ◴[] No.35474351[source]
The design of dynamic linking on most *nix-ish systems today comes from SunOS in 1988, and doesn't have much to do with C at all other than requiring both the compiler and assembler to know about position-independent code.

What elements of dynamic linking do you see as being connected to "70s era programming languages"?

> Yes, I considered that possibility.

Then I would urge you to reconsider.

replies(2): >>35475191 #>>35479652 #
preseinger ◴[] No.35475191[source]
dynamic linking is an optimization that is no longer necessary

there is no practical downside to a program including all of its dependencies, when evaluated against the alternative of those dependencies being determined at runtime and based on arbitrary state of the host system

monomorphization is good, not bad

the contents of /usr/lib/whatever should not impact the success or failure of executing a given program

replies(1): >>35475419 #
PaulDavisThe1st ◴[] No.35475419[source]
Dynamic linking wasn't an optimization (or at least, it certainly wasn't just an optimization). It allows for things like smaller executable sizes, more shared code in memory, and synchronized security updates. You can, if you want, try the approach of "if you have 384GB of RAM, you don't need to care about these things", and in that sense you're on quicksand with the "just an optimization". Yes, the benefits of sharing library code in memory are reduced by increasing system RAM, but we're seeing from a growing chorus of both developers and users, the "oh, forget all that stupid stuff, we've got bigger faster computers now" isn't going so well.

There's also the problem that dynamic loading relies on almost all the same mechanisms as dynamic linking, so you can't get rid of those mechanisms just because your main build process used static linking.

replies(1): >>35476228 #
preseinger ◴[] No.35476228[source]
it allows for all of the things you list, yes, but those things just aren't really valuable compared to the reliable execution of a specific binary, regardless of any specific shared library that may be installed on a host

smaller executable sizes, shared code in memory, synchronized security updates, are all basically value-zero, in any modern infrastructure

there is no "growing chorus" of developers or users saying otherwise, it is in fact precisely the opposite, statically linked binaries are going extremely well, they are very clearly the future

replies(2): >>35476415 #>>35478228 #
PaulDavisThe1st ◴[] No.35476415[source]
the chorus is about the assumptions commonly found among younger devs that these old "efficiency" and "optimization" techniques don't matter any more. c.f. apps (desktop, mobile) that take forever to do things that should not take forever.

"modern infrastructure" seems like a bit of a giveaway of your mind set. yes, i know that there's a lot of stuff that now happens by having your web browser reach out to "infrastructure" and then the result is displayed in front of you.

But lots of people still use their computers to run applications outside the browser, where "modern infrastructure" means either nothing at all, or it means "their computer (or mobile platform)". the techniques mentioned in this subthread are all still very relevant in this context.

replies(1): >>35476982 #
preseinger ◴[] No.35476982[source]
there is basically no situation in which it is important to optimize for binary size, embedded sure, but nowhere else

the infrastructural model i'm describing doesn't require applications to run in browsers, or imply that applications are slower, actually quite to the contrary, statically linked binaries tend to be faster

the model where an OS is one to many with applications works fine for personal machines, it's no longer relevant for most servers (shrug)

replies(3): >>35478013 #>>35478165 #>>35550312 #
1. rektide ◴[] No.35550312[source]
binary size is also memory size. memory size matters. applications sharing the same libraries can be a huge win for how much stuff you can fit on a server, and that can be a colossal time/money/energy saver.

yes: if you're a company that tends to only run 1-20 applications, no, the memory-savings probably won't matter to you. that matches quite a large number of use cases. but a lot of companies run way more workloads than anyone would guess. quite a few just have no cost-control and/or just don't know, but there's probably some pretty sizable potential wins. it's even more important for hyper-scalers, where they're running many many customer processes at a time. even companies like facebook though, i forget the statistic, but sometime in the last quarter there was quote saying like >30% of their energy usage was just powering ram. willing to bet, they definitely optimize for binary size. they definitely look at it.

there's significant work being put towards drastically reducing scale of disk/memory usage across multiple containers, for example. composefs is one brilliant very exciting example that could help us radically scale up how much compute we can host. https://news.ycombinator.com/item?id=34524651

i also haven't seen the very important very critical other type of memory mentioned, cache. maybe we can just keep paying to add DRAM forever and ever (especially with CXL coming across the horizon), but the SRAM in your core-complex will almost always tend to be limited (although word is Zen4 might get within striking distance of 1GB which is EPIC). static builds are never going to share cache effectively. the instruction cache will always be unique per process. the most valuable expensive fancy memory on the computer is totally trashed & wasted by static binaries.

there's really nothing to recommend about static binaries, other than them being extremely stupid. them requiring not a single iota of thought to use is the primary win. (things like monomorphic optimization can be done in dynamic libraries with various metaprogramming & optimizing runtimes, hopefully one's that don't need to keep respawning duplicate copies ad-nauseum.)

i do think you're correct about the dominant market segment of computing, & you're speaking truthfully to a huge % of small & mid-sized businesses, where the computing needs are just incredibly simple & the ratio of processes to computers is quote low. their potential savings are not that high, since there's just not that much duplicate code to keep dynamically linking. but i also think that almost all interesting upcoming models of computing emphasize creating a lot more smaller lighter processes, that there are huge security & managability benefits, and that there's not a snowman's chance in hell that static-binary style computing has any role to play in the better possible futures we're opening up.

replies(1): >>35552509 #
2. preseinger ◴[] No.35552509[source]
you're very sensitive to the costs of static linking but i don't think you see the benefit

the benefit is that a statically linked binary will behave the same on all systems and doesn't need any specific runtime support above or beyond the bare minimum

this is important if you want a coherent deployment model at scale -- it cannot be the case that the same artifact X works fine on one subset of hosts, but not on another subset of hosts, because their openssl libraries are different or whatever

static linking is not stupid, it doesn't mean that hosts can only have like 10 processes on them, it doesn't imply that the computing needs it serves are simple, quite the opposite

future models of computing are shrinking stuff like the OS to zero, the thing that matters is the application, security (in the DLL sense you mean here) is not a property of a host, it's a property of an application, it seems pretty clear to me that static linking is where we're headed, see e.g. containers