Bcachefs may be headed out of the kernel

bcachefs is experimental and Kent writes in the LWN comments that nothing would get done if he didn't develop it this way. Filesystems are a massive undertaking and you can have all the rules you want. It doesn't help if nothing gets developed.

It would be interesting how strict the rules are in the Linux kernel for other people. Other projects have nepotistic structures where some developers can do what they want but others cannot.

Anyway, if Linus had developed the kernel with this kind of strictness from the beginning, maybe it wouldn't have taken off. I don't see why experimental features should follow the rules for stable features.

If it's an experimental feature, then why not let changes go into the next version?

That is a valid objection, but I still think that for some huge and difficult features the month long pauses imposed by release cycles are absolutely detrimental.

Ideally they'd be developed outside the kernel until they are perfect, but Kent addresses this in his LWN comment: There is no funding/time to make that ideal scenario possible.

He could release a patch that can be pulled by the people that need it.

If you’re using experimental file systems, I’d expect you to be pretty competent in being able to hold your own in a storage emergency, like compiling a kernel if that’s the way out.

This is a made up emergency, to break the rules.

This position seems so incoherent. If it’s so experimental, why is it in the mainline kernel? And why are fixes so critical they can’t wait for a merge window? Who is using an “experimental” filesystem for mission-critical work that also has to be on untested bleeding-edge code?

Like the sibling commenter, I suspect the word “experimental” is being used here to try and evade the rules that, somehow, every other part of the kernel manages to do just fine with.

No, you have to understand that filesystems are massive (decade+) projects, and one of the key things you have to do with anything that big that has to work that perfectly is a very gradual rollout, starting with the more risk tolerant users and gradually increasing to a wider and wider set of users.

We're very far along in that process now, but it's still marked as experimental because it is not quite ready for widespread deployment by just anyone. 6.16 is getting damn close, though.

That means a lot of our users now are people getting it from distro kernels, who often have never compiled a kernel before - nevertheless, they can and do report bugs.

And no matter where you are in the rollout, when you get bug reports you have to fix them and get the fixes out to users in a timely manner so that they can keep running, keep testing and reporting bugs.

It's a big loss if a user has to wait 3 months for a bugfix - they'll get frustrated and leave, and a big part of what I do is building a community that knows how the system works, how to help debug, and how to report those bugs.

A very common refrain I get is "it's experimental <expletive deleted>, why does it matter?" - and, well, the answer is getting fixes out in a timely manner matters just as much if not more if we want to get this thing done in a timely manner.

Isn't this the point of DKMS, to decouple module code from kernel code?

Well, my hope when bcachefs was merged was for it to be a real kernel community project.

At the time it looked like that could happen - there was real interest from Redhat prior to merging. Sadly Redhat's involvement never translated into much code, and while upstreaming did get me a large influx of users - many of which have helped enormously with the QA and stabilization effort - the drama and controversies have kept developers away, so on the whole it's meant more work, pressure and stress for me.

So DKMS wouldn't be the worst route, at this point. It would be a real shame though, this close to taking the experimental label off, and an enormous hassle for users and distributions.

But it's not my call to make, of course. I just write code...

DKMS is an awful user experience, it's an easy way to render a system unbootable. I hope Linus doesn't force existing users, like me, down that path. It's why I avoid zfs, however good it may be.

I am sympathetic to your plight. I work on internal dev tools and being able to get changes out to users quickly is an incredible tool to be able serve them well and earn (or keep) their trust.

It seems like you want that kind of fast turn around time in the kernel though, which strikes me as an impossible goal.

> evade the rules that, somehow, every other part of the kernel manages to do just fine with

I have no context on the situation or understanding of what the right set of rules here is, but the difference between filesystems and other code is that bugs in the filesystem can cause silent, persistent corruption for user data, on top of all the other failure modes. Most other parts of the kernel don't have such a large persistence capability in case of failure. So I can understand if filesystems feel the need to be special.

Yet the other filesystems seem fine with the rules. And the value proposition of Bcachefs precisely is that it doesn't eat your data. So, either the marketing is off, or it is far from ready to live with the quite predictable release pace of the Linux kernel.

DKMS isn't a "fire and forget" kind of tool, but it comes reasonably close most of the time. I would say it's a far cry from awful, though.

IMHO, anybody willing using a file system marked as experimental from a downstream kernel should be able to wait for the fix. If they need it faster they should be ready to compile their own kernel or seriously reevaluate their decision to adopt the FS.

The Kernel's pace is predictable, even billion $ corporates can live with it, and it's not like Linus hasn't accommodated you in the past. But continuing to do so will make people stop believing you are acting in good faith, and the outcome of that will be predictable.

This is simply how the development model is like in the Linux kernel. Exceptions happen, but they sometimes backfire and highlight why the rules matter in the first place, and therefore they are unlikely to change.

One of my machines runs root on ZFS via DKMS. I will grant that it is annoying, and it used to be worse, but I don't think it's been quite as bad as all that for a very long time. I would also argue that it's more acceptable for testing actively developed stuff that's getting the bugs worked out in order to work towards mainlining.

That said, I vaguely seem to recall that bcachefs ended up involving changes to other parts of the kernel to better support it; if that's true then DKMS is likely to be much more painful if not outright impossible. It's fine to compile one module (or even several) against a normal kernel, but the moment you have to patch parts of the "main" kernel it's gonna get messy.

The inconvenience of this process is also addressed by the dev, as is the different definition of experimental that you're using (though your expectation re kernel doesn't follow even without the mismatch in definitions)

See the replies made by Josef Bacik and Theodore Ts'o.

https://lwn.net/ml/all/20250627144441.GA349175@fedora/#t

https://lwn.net/ml/all/20250628015934.GB4253@mit.edu/

My impression as a total outsider here is that most (all?) other filesystems I'm aware of are either more mature - and generally not in active feature development - or they are not as popular, limiting the damage. Is this inaccurate?

I will also say that bcachefs's selling point - and probably a major reason people are so excited for it - is amount of effort it puts into avoiding data corruption. Which tells you something about the perceived quality of other filesystems on Linux. Which means that saying "other filesystems seem fine with the rules" misses the very fact that people have seen too much data corruption with other filesystems and want something that prioritizes it higher.

> Kent writes in the LWN comments

Unfortunately Kent spends a lot of time and effort defending Kent. I wish he would learn to take a step back and admit he's fallible and learn to play nice in the sandbox rather than wasting all of this time and effort. A simple "mea culpa" could smooth over a lot of the feathers he constantly ruffles.

> It's a big loss if a user has to wait 3 months for a bugfix

Either the bugfix is not serious and they can wait because the system is mature. Or, The fs is so unstable you shouldn't be pandering to the crowd that struggle with build deps and make install.

There is no in between, this is the situation. And the "but not all bugs are equal" argument doesn't stand.

I know if I read of a metadata but getting fixed in ext4 or ZFS there's a very small chance of this causing my platter to evaporate. By definition of stable, if that was happening it would be hitting that one unfortunate guy (<0.001% of users) running a weird OS or hardware and that's just the luck of the draw.

If the fix is from a fs marked experimental, yes I kinda expect this could fry my data and hurt kittens. That's what experimental and stable mean. That means I expect up to 100% of users to be impacted under certain workflows or scenarios.

Everything outside of this is wasted energy and arguing with the tide.

Btrfs is still very much being developed, in the kernel and is quite popular.

ZFS should be avoided because it has too many dumb complete failure states (having run it in a real production storage environment), not because it's DKMS

> It's a big loss if a user has to wait 3 months for a bugfix

Is the wait really 3 monts away? I don't exactly know the release cycle, but for me kernels are released quite frequently, at least there are RC sooner than 3 months. Just checked latest history and major releases are 2 months apart - and between them there are minor ones.

People using experimental features are quite aware how to get new experimental kernel sources.

Chris Mason moved on a long time ago, Josef seems to be spending most of his time on other things, and if you look at the commit history btrfs development has been moving pretty slowly for a long time.

It's a bad sign when the key founders leave like that, filesystems require a lot of coherence of design and institutional knowledge to be retained.

DKMS as an option might be better then you imagine.

The kernel, even its bugs, should be stable (in that they shouldn't change unless it happens the correct way). If not, it starts introducing unexpected issues to users.

If someone's testing against these versions, adding their fixes and patches, stuff like this will break things for users. He can't assume all users will be regular desktop users, even on an experimental area of the code.

Things like 'RC' have meaning. Meaning that has been there for years. He can develop on a separate tree and users that want it can use it. This is used all over.

> That is a valid objection, but I still think that for some huge and difficult features the month long pauses imposed by release cycles are absolutely detrimental.

I feel you're not answering the question, nor are you presenting any case in favor of forcing an exceptional release process for an unstable feature.

The "detrimental" claim is also void of any reason or substance. It's not to it's users as users know better than rolling out experimental file systems for critical data, and those hypothetical users who somehow are really really interested in using bleeding edge will already be building their own kernel for this reason alone. Both scenarios don't require this code to be in the kernel, let alone exceptional release processes.

> Ideally they'd be developed outside the kernel until they are perfect, but Kent addresses this in his LWN comment: There is no funding/time to make that ideal scenario possible.

It is clear then that the code should be pulled from the kernel. If it isn't impossible to maintain a feature with the regular release process, everyone stands to benefit by not shipping code that is impossible to stabilize.

> The inconvenience of this process is also addressed by the dev, as is the different definition of experimental that you're using (...)

The only aspect of "experimental" that matters is what it means to the release process. If you can't meet that bar then debating semantics won't help either.

And by the way, the patch thread clearly stresses a) loss of data, b) the patch trying to sneak under the maintenance radar new features. That is the definition of unstable in anyone's book.

I think my problem is that it's just close enough to being fire-and-forget that I forget how to do the recovery when it misfires. It usually seems to crop up when I'm on vacation or something and I don't have my tools.

> The "detrimental" claim is also void of any reason or substance.

Thanks for the compliments! Detrimental for development speed, not for the users.

I’ve run racks and racks of it in prod also. What are these dumb complete failure states you mean?

Experimental has no defined meaning with respect to the release process.

It's a signal to users - "watch out if you're going to use this thing"

> It's a big loss if a user has to wait 3 months for a bugfix

This is incredibly short-sighted. You're talking about 1 user 3 months, and you think that's "big" ? I'd say it's a much bigger loss if the project gets kicked out because of one person's impatience. Then everybody will have to wait forever, how is that better?

If the fs is as good as you claim, then you better play by the rules and make sure the project survives and eventually goes GA. If it happens a few months later, then so be it. Think about the long term.

If you're worried about a single user leaving, then a much better strategy would be to explain to this user the Linux release timeline, or how to apply a patch, than to go up toe to toe against Linus.

And btw, squeezing a fix/feature in at the last minute in order to help one user is not as good as you think it is. Even if that one user appreciates your responsiveness, to everyone else it sends a message that the key dev is super impatient and unprofessional. So even if you manage to keep that one user, how many potential users are you losing by sending that message?