The patch that kicked off the current conflict was the 'journal_rewind' patch; we recently (6.15) had the worst bug in the entire history upstream - it was taking out entire subvolumes.
The third report got me a metadata dump with everything I needed to debug the issue, thank god, and now we have a great deal of hardening to ensure a bug like this can never happen again. Subsequently, I wrote new repair code, which fully restored the filesystem of the 3rd user hit by the bug (first two had backups).
Linus then flipped out because it was listed as a 'feature' in the pull request; it was only listed that way to make sure that users would know about it if they were affected by the original bug and needed it. Failure to maintain your data is always a bug for a filesystem, and repair code is a bugfix.
In the private maintainer thread, and even in public, things went completely off the rails, with Linus and Ted basically asserting that they knew better than I do which bcachefs patches are regression risks (seriously), and a page and a half rant from Linus on how he doesn't trust my judgement, and a whole lot more.
There have been many repeated arguments like this over bugfixes.
The thing is, since then I started perusing pull requests from other subsystems, and it looks like I've actually been more conservative with what I consider a critical bugfix (and send outside the merge window) than other subsystems. The _only_ thing that's been out of the ordinary with bcachefs has been the volume of bugfixes - but that's exactly what you'd expect to see from a new filesystem that's stabilizing rapidly and closing out user bug reports - high volume of pure bugfixing is exactly what you want to see.
So given that, I don't think having a go-between would solve anything.
1. Regardless of whether correct or not, it's Linus that decides what's a feature and what's not in Linux. Like he has for the last however many decades. Repair code is a feature if Linus says it is a feature.
2. Being correct comes second to being agreeable in human-human interactions. For example, dunking on x file system does not work as a defense when the person opposite you is a x file system maintainer.
3. rules are rules, and generally don't have to be "correct" to be enforced in an organization
I think your perceived "unfairness" might make sense if you just thought of these things as un-workaroundable constraints, Just like the fact that SSDs wear out over time.
Do you argue with your school teachers that your book report shouldn't be due on Friday because it's not perfect yet?
I read several of your response threads across different websites. The most interesting to me was LWN, about the debian tools, where an actual psychologist got involved.
All the discussions seem to show the same issue: You disagree with policies held by people higher up than you, and you struggle with respecting their decisions and moving on.
Instead you keep arguing about things you can't change, and that leads people to getting frustrated and walking away from you.
It really doesn't matter how "right" you may be... not your circus, not your monkeys.
Edit since you expanded your post:
>The most interesting to me was LWN, about the debian tools, where an actual psychologist got involved.
To me the comment was patronizing implying it was purely due to bad communication from Kent's end and shows how immature people are with running these operating system are. Putting priority on processes over the end user.
>respecting their decisions and moving on.
When this causes real pain for end users. It's validating that the decision was wrong.
> really doesn't matter how "right" you may be... not your circus
It does because it causes reputational damage for bcachefs. Even beyond reputational damage, delivering a good product to end users should be a priority. In my opinion projects as big as Debian causing harm to users should be called out instead of ignored. Else it can lead to practices like replacing dependencies out from underneath programs to become standard practice.
For many humans to work together over time on something very complex is hard. Structure and process are required. And sometimes they come at the expense of what some might call “pure” engineering. But they are the right trade offs to optimize for the actual goal.
If you can’t accept that, stick to solo projects.
This is the difference between being smart and being wise. If the goal of all this grandstanding was that, it's so incredibly and vitally important for these patches to get into the kernel, well guess what, now due to all this drama this part of the kernel is going to go unmaintained entirely. Is that good for the users? Did that help our stated goal in any way? No.
The adult thing is to do best by the users. Critical file system bugs are worth blocking the release of any serious operating system in the real world as there is serious user impact.
>Is that good for the users?
I think it's complicated. It could allow for a faster release schedule for bug fixes which can allow for addressing file system issues faster.
Best by users in the long term is predictable processes. "RC = pure bug fixes" is a battle tested, dependable rule, absence of which causes chaos.
> Critical file system bugs are worth blocking the release
"Experimental" label EXACTLY to prevent this stuff from blocking release. Do you not know that bcachefs is experimental? This is an example of another rule which helps predictability.
Prioritizing agreeableness above correctness is the reason the space shuttle Challenger blew up.
The bcachefs fracas is interesting and important because it's like a stain making some damn germ's organelles visible: it highlights a psychological division in tech and humanity in general between people who prioritize
1) deferring to authority, reading the room, knowing your place
and people who prioritize
2) insisting on your concept of excellence, standing up against a crowd, and speaking truth to power.
I am disturbed to see the weight position #1 has accumulated over the past decade or two. These people argue that Linus could be arbitrarily wrong and Overstreet arbitrarily right and it still wouldn't matter because being nice is critical to the success of a large scale project or something.
They get angry because they feel comfort in understanding their place in a social hierarchy. Attempts to upend that hierarchy in the name of what's right creates cognitive dissonance. The rule-followers feel a tension they can relieve only by ganging up and asserting "rules are rules and you need to follow them!" --- whether or not, at the object level, a) there are rules, b) the rules are beneficial, and c) whether the rules are applied consistently. a, b, and c are exactly those object-level does-the-o-ring-actually-work-when-cold considerations that the rule-following, rule-enforcing kind of person rejects in favor a reality built out of words and feelings, not works and facts.
They know it, too. They need Overstreet and other upstarts to fail: the failure legitimizes their own timid acquiescence to rules that make no sense. If other people are able to challenge rules and win, the #1 kind of person would have to ask himself serious and uncomfortable questions about what he's doing with his life.
It's easier and psychologically safer to just tear down anyone trying to do something new or different.
The thing is all technological progress depends on the #2 people winning in the end. As Feynmann talked about when diagnosing this exact phenomenon as the root cause of the Challenger disaster, mother nature (who appears to have taken on corrupting filesystems as a personal hobby of hers) does not care one bit about these word games or how nice someone is. The only thing that matters when solving a problem of technology is whether something works.
I think a lot of people in tech have entirely lost sight of this reality. I can't emphasize enough how absurd it is to state "[b]eing correct comes second to being agreeable in human-human interactions" and how dangerously anti-technology, anti-science, and-civilization, and anti-human this poison mindset is.
Didn't Linus lambast you for "lack of testing and collaboration before submitting patches", to the point the patches you were trying to push weren't even building?
https://ostechnix.com/linus-torvalds-expresses-frustration-w...
I think it's less subtle than that. The straw that broke the camel's back was quite literally abuse towards other kernel developers.
>"Experimental" label EXACTLY to prevent this stuff from blocking release
In practice bcachefs is used in production with real users. If the experimental label prevents critical bug fixes from making it into the kernel then it would be better to just remove that label.
I don't know the situation well enought to review where they drew the line, but there definitely should be a line somewhere.
I'm not sure exactly what you are talking about, and I'm not sure you do either. The discussion that preceded bcachefs to be dropped from the Linux kernel mainline involved an attempt to sneak a new features in RC, sidestepping testing and QA work, which was followed up by yet more egregious behavior from the mantainer.
https://www.phoronix.com/news/Linux-616-Bcachefs-Late-Featur...
Too solve a bug with the filesystem that people in the wild were hitting. Like how Linus has said in the past with how there is a blurry line between security fixes and bug fixes. There is a blurry line between filesystem bugs and recovery features.
If you read the email it is clear that the full feature has more work needed and this is more of a basic implementation to address bugs that people hit in the wild.
alternative perspective: those users have knowingly and willingly put experimental software into production. it was their choice, they were informed of the risk and so the consequences and responsibility are their’s.
it’s like signing up to take some experimental medicine, and then complaining no-one told me about the side-effect of persistent headaches.
that doesn’t stop anyone from being user-centric in their approach, e.g. call me if you notice any symptoms and i’ll come round your house to examine you.
… as long as everyone is clear about the fact it is experimental and the boundaries/limitations that apply, e.g. there will be certain persistent headache medicines that cannot be prescribed to you, or it might take longer for them to work because you’re on an experimental medicine.
Linus doesn't seem to believe in automated testing. He just seems to think that there's no way I could QA code as quickly as I do, but that's because I've invested heavily in automated testing and building up a community of people doing very good testing and QA work; bcachefs's automated testing is the best of any upstream filesystem that I've seen (there's a whole cluster of machines dedicated to this), and I have people running my latest branch on a daily basis.
Nearly all of the collaboration just happens on IRC.
For big changes I wait for explicit acks from testers that they've ran it and things look good; a lot of people read and review my code too, it's just typically less formal than the rest of the kernel.
So you acknowledge that this last episode involved trying to push new features into a RC.
As it was made abundantly clear, not only is the point of RC branches to only get tiny bugfixes after testing, the feature work that was presented was also untested and risked introducing major regressions.
All these red flags were repeatedly raised in the mailing list by multiple kernel maintainers. Somehow you're ignoring all the feedback and warnings and complains raised by people from Linux kernel maintainers, and instead you've opted to try to gaslight the thread.
All your comments are dismissive of the criticisms so far and you’re shrugging your shoulders as to why.
It’s great you’re able to reason and defend yourself but Linux as a whole is larger than you and refusing to submit to their ways will make technology move no where.
This puts us all in a shitty situation. I want the experimental label to come off at the right time - when every critical bug is fixed and it's as trustworthy as I can reasonably make it, when I know according to the data I have that everyone is going to have a good experience - but I have real users who need this thing and need to be supported.
There is _no reason_ to interpret the experimental label in the way that you're saying, you're advocating that reliability for the end user be deprioritized versus every other filesystem.
But deprioritizing reliability is what got us into this mess.
Excellent engineering management largely isolates engineers from having to deal with this non-engineering stuff (except for the subset that is specifically for their own personal benefit)-- but open source tends to radically flatten organizations that produce software, such that every contributor must also be their own manager to a great degree.
In a well run project you don't necessarily have to be good at or even interested in all the more socially oriented components of the project organization. But if you're not you must be willing to let someone else handle that stuff and go along with their judgements even if they seem suboptimal from the narrower perspective you've adopted. If you can't then from a "collaborative development as a system" view you're a faulty component that doesn't provide the right interface for the system's requirements (and are gonna get removed!). :)
Another way to look at it is that it would be ideal if every technical element were optimal at all times. In small systems with well understood requirements this can be possible or at least close to possible. But in big complex and poorly scoped systems it's just not possible: We have imperfect information, there are conflicting requirements, we have finite time, and so on. The system as a whole will always be far from perfect. If anyone tried to make it all perfect it would just fail to make progress, deadlock, or otherwise. The management of the project is always trying to balance the imperfections. They know that their decisions are often making things worse for a local concern, but they do so with belief that over time the decisions result in a better system overall. Linux has a good reputation in large part due to a long history of making good decisions about the flaws to accept or even introduce, which issues to gloss over vs debate to death.
bcachefs has a ton of QA, both automated testing and a lot of testers that run my latest and I work with on a daily basis. The patch was well tested; it was for codepaths that we have good regression tests for, it was algorithmically simple, and it worked perfectly to recover a filesystem from the original bug report, and it performed flawlessly again not long after.
I've explained my testing and QA on the lists multiple times.
You, like the other kernel maintainers in that thread, are making wild assertions despite having no involvement with the project.
When you're working on the core technology we all depend on, correctness is not optional.
1. I laid down what I perceived as the state of things. The generalizations I drew from observing the system that is Linux development. Nowhere have I prescribed that kent "follow" my ideas. Simply that he can use these to try to understand the unfairness he feels.
2. Your anarcho-individualistic development ideas sound good in theory, but if they ever worked in practice we might have seen it be more widespread than it is today in team sizes > 3.
You should also note that if the oring is labelled experimental and there's an expectation of failure, it's development and testing will not stop the launch. The shuttle leaves when it leaves, it won't wait for the experimental oring to be done to your liking.
You must know this when it comes to your own work. Why isn't bcachefs written in augmented rust with dependent types and formal correctness proofs for every line of code? How could there ever be a data losing bug if you had a formal proof that the file system could never lose data? Wouldn't that be more correct?
Turns out when some strong/broad notion of correctness isn't (practically) possible it is, in fact, very optional.
Good project management is all about managing resources and balancing tradeoffs. Sometimes this means making or allowing some things to be worse for the benefit of something else or in adherence to a process with a proven track record. Almost every choice makes something less correct than it could be-- with a goal of slowly inching towards a more perfect state overall in the long run.
It's also beneficial to rock the boat a bit at times, people can be wrong, processes can need improvement-- but there is a correct level, timing, and approach to achieve the best benefit. I expect that the kind of absolute approach you seem to have adopted in comments is unlikely to be successful at effective beneficial change.
It sounds like you have a hard time coping with reality.
https://www.phoronix.com/news/Linux-616-Bcachefs-Late-Featur...
I repeat: it sounds an awful lot like you are trying to gaslight this thread. Not cool.
When this fact was again explicitly pointed out to you by Linus himself, you even tried to bullshit Linus and try to move the goalpost with absurd claims about how somehow it was ok to force untested and unreviewed features into a RC because somehow you know better about what users want or need as if it was some kind of justification for you to skip testing and proper release processes.
You need to set aside some time for introspection because you sound like you are your own worst enemy. And those you interact with seem to be fed up and had enough of these stunts.
Even taking your claims at face value (which from this thread alone is a heck of a leap) I'm baffled by the way you believe this holds any relevance.
I mean, the kernel project has in place a quality assurance process designed to minimize the odds of introducing problems when preparing a release. You were caught purposely ignoring any QA process in place and trying to circumvent the whole quality assurance process and sneak into a RC features that were untested and unverified.
There is a QA process, and you purposely decided to ignore it and plow away. And then your best argument for purposely ignoring any semblance of QA is that others may or may not have broken a build before?
Come on, man. You know better than this. How desperate are you to avoid any accountability to pull these gaslighting stunts?
You're suggesting he deal with unfairness by internalizing it as virtue? That's how to make people who cheer at other people's failures.
> Your anarcho-individualistic development ideas sound good in theory
Thanks for illustrating my point. No project, >3 or <= 3, has ever made any new technology by adopting as a tenet that social agreement inside the project is more important than correctly modeling the world outside it, and you're suggesting I'm using inefficiently agreeable-sounding words to express it.
I don't buy it. Object level considerations do matter. Alice's bug really is worse than Bob's. That "proven track record" shouldn't apply to Alice, and insisting that it does for the sake of process, in a way indifferent to the facts of the situation, is just a pretext for doing primate social hierarchy deference rituals in a situation in which they're producing a worse outcome and everyone knows it.
PLEASE, honestly, EDUCATE THESE USERS. This is still marked experimental for numerous reasons regardless of the 'planned work for 6.18'. Users who can't suffer any data loss and are repeating their mistake of using btrfs shouldn't be using a none default/standard/hardened filesystem period.
They do. And Kent expressed them and the linux kernel maintainers are amply qualified to hear out and make a call. I don't see a reason to think they were indifferent to the facts, they just weren't convinced by them. If they were they could have just said, "okay we think that this does qualify as a bugfix".
My understanding is the change in dispute wasn't over fixing the corruption introducing bug, but rather adding automated repair for cases where the corruption had already happened. I could easy see taking a position of "sad for people who are already corrupt, they can get their work around out of tree for now" (or heck, even forever depending on the scale of the impact).
Anyone who has been around for a while has seen their share of 'ate the horse to catch the spider to catch the fly to...' dance, of course the patch author is convinced that their repair is correct. They're almost always convinced of that or they don't submit it, so that carries little information. Because of this there is a strong preference for obviously minimal code in any kind of fix. Minimizing user suffering is important, but we also know every line of code comes with risk. The fact that the risk is not measurable on a case by case basis doesn't make it any less real.
Oh dear lord no. That is not even what _any_ of the actual investigations suggested.
woke agreeableness is bad but it wasn't getting along at the water-cooler that lead to challenger.
You seem careful not to refer to any of his decisions as your own, but the writing style and inability to respect authority is still there.
In the past I've often told people who wanted to migrate off of btrfs "check back in six months", but I'm not now because 6.16 is looking amazingly solid; all the data I have says that your data really is safer on bcachefs than btrfs.
I'm not advocating for people to jump from ext4/xfs/zfs, that needs more time.
> I don't see a reason to think they were indifferent to the facts
I don't think the Linux people thought of themselves as indifferent to facts. Nor do I think they were, not at first. Most people imagine themselves as fair-minded truth-seekers. When stakes are low, they usually act like it. It's only under pressure that people reveal whether they're more committed to PR or progress.
The shitty thing about this situation is that as the dispute escalated, the technical merits of change faded from relevance. (Linus even pulled the corruption repair work in the end!) The argument transformed into a dispute over power, pride, and personalities. Linus's commitment to technical excellence was tested. It failed. Consequently, Linux will lack a cutting-edge filesystem.
I don't even object to Linus being BDFL of Linux. Somebody has to make decisions. I think Linus was wrong to reject the corruption fix patch, but he could plausibly have been right. He had an opportunity to explain his patch rejection in such a way that Overstreet would have understood it as final but also felt heard and valued. Overstreet would have been upset, and justifiably so, but by the next merge window both sides would have cooled down and progress would have resumed.
It's when Linus banned Overstreet and bcachefs from the project that he departed irrecoverably from defensibility. Linus might think he's punishing Overstreet for his intransigence by blocking his work, but Linus is actually taking his frustration out on every Linux user instead. Overstreet's ban is rooted in primate power psychology, not technical trade-offs, and it makes everyone lose.
Technical leaders who ostracize brilliant but difficult people forever cap the amount of progress we can make in the fight against the limits of nature. They're neglecting their responsibilities as leaders to harness difficult people. It's not an easy job, but being a leader shouldn't be.
Linus took the easy way out and banned the brilliant troublemaker. He should be ashamed.
> the risk is not measurable on a case by case basis
It often is. That's why when I'm on the Linus side of a case like this, I try to avoid saying "no" and instead say "yes, if". Sometimes my counterparty pulls out an "if" that convinces me.
Sorry, the only person gaslighting here is you.
Don't compare bcachefs with btrfs for stability. Compare it with ext4. (And dont care anecdotal data, compare the process).