←back to thread

SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

(www.source.dev)
137 points cdesai | 5 comments | 22 Oct 25 12:39 UTC | HN request time: 0.005s | source
Show context
Ericson2314 ◴[22 Oct 25 15:46 UTC] No.45670912[source]▶
The headline times are a bit ridiculous. Are they trying to turn https://github.com/facebook/sapling/blob/main/eden/fs/docs/O... or some git fuse thing into a product?
replies(2): >>45671058 #>>45671060 #
zokier ◴[22 Oct 25 15:55 UTC] No.45671060[source]▶
Well they also claim to be able to cache build steps somehow build-system independently.

> As the build runs, any step that exactly matches a prior record is skipped and the results are automatically reused

> SourceFS delivers the performance gains of modern build systems like Bazel or Buck2 – while also accelerating checkouts – all without requiring any migration.

Which sounds way too good to be true.

replies(4): >>45671827 #>>45671987 #>>45672236 #>>45677349 #
vlovich123 ◴[22 Oct 25 16:57 UTC] No.45671987[source]▶
Yeah, I agree. This part is hand waved away without any technical description of how they manage to pull this off since knowing what is even a build step and what dependencies and outputs are are only possible at the process level (to disambiguate multi threaded builds). And then there’s build steps that have side effects which come up a lot with CMake+ninja.
replies(1): >>45673038 #
1. rcxdude ◴[22 Oct 25 18:14 UTC] No.45673038[source]▶
A fuse filesystem can get information about the thread performing the file access: https://man.openbsd.org/fuse_get_context.3

So they could in principle get a full list of dependencies of each build step. Though I'm not sure how they would skip those steps without having an interposer in the build system to shortcut it.

replies(3): >>45674200 #>>45674223 #>>45674474 #
2. vlovich123 ◴[22 Oct 25 19:45 UTC] No.45674200[source]▶
Yeah that’s what I meant. I bet you the build must be invoked through a wrapper script that interposes all executables launched within the product tree. Complicated but I think it could work. Skipping steps correctly is the hard part but maybe you do that in terms of knowing somehow the files that will be accessed ahead of time by that processes and then skipping the launch and materializing the output (they also mention they have to run it once in a sandbox to detect the dependencies). But still, side effects in build systems seem difficult to account for correctly; I bet you that’s why it’s a “contact us” kind of product - there’s work needed to make sure it actually works on your project.
3. mook ◴[22 Oct 25 19:47 UTC] No.45674223[source]▶
Didn't tup do something like that? https://gittup.org/tup/index.html Haven't looked at it in a while, no idea if it got adoption.

But initially the article sounded like it was describing a mix of tup and Microsoft's git vfs (https://github.com/microsoft/VFSForGit) mushed together. But doing that by itself is probably a pile of work already.

replies(1): >>45675075 #
4. ◴[22 Oct 25 20:07 UTC] No.45674474[source]▶
5. serbancon ◴[22 Oct 25 20:59 UTC] No.45675075[source]▶
Yes, you are correct - SourceFS also caches and replays build steps in a generic way. It works surprisingly well, to the point where it’s hard to believe until you actually see it in action (here is a short demo video, but it probably isn't the best way to showcase it: https://youtu.be/NwBGY9ZhuWc?t=76 ).

We intentionally kept the blog post light on implementation details - partly to make it accessible to a broader audience, and partly because we will be posting gradually some more details. Sounds like build caching/replay is high on the desired blogpost list - ack :-).

The build-system integration used here was a one-line change in the Android build tree. That said, you’re right - deeper integration with the build system could push the numbers even further, and that’s something we’re actively exploring.