Most active commenters
  • rwmj(4)
  • arp242(3)
  • koverstreet(3)

←back to thread

214 points ksec | 37 comments | | HN request time: 1.267s | source | bottom
1. sevg ◴[] No.45076556[source]
Is it just me or does Kent seem self-destructively glued to his own idea of how kernel development should work?

I don’t doubt that people on all sides have made mis-steps, but from the outside it mostly just seems like Kent doesn’t want to play by the rules (despite having been given years of patience).

replies(5): >>45077241 #>>45077371 #>>45077492 #>>45077724 #>>45080172 #
2. toast0 ◴[] No.45077241[source]
It's not just kernel development. In the lwn thread, he mentioned and then demonstrated difficulty working with Debian developers as well.

IMHO, what his communications show is an unwillingness to acknowledge that other projects that include his work have focus, priorities, and policies that are not the same as that of his project. Also, expecting exceptions to be made for his case, since exceptions have been made in other cases.

Again IMHO, I think he would be better off developing apart with an announcement mailing list. When urgent changes are made, send to the announcement list. Let other interested parties sort out the process of getting those changes into the kernel and distributions.

If people come with bug reports from old versions distributed by others, let them know how to get the most up to date version from his repository, and maybe gently poke the distributors.

Yes, that means users will have older versions and not get fixes immediately. But what he's doing isn't working to get fixes to users immediately either.

3. ajb ◴[] No.45077371[source]
I think Kent is in the wrong here, but it really doesn't help that the kernel people from Linus on down are seemingly unable to explain the problem, and instead resort to playground insults. Apart from being unprofessional and making for a hostile work environment, it doesn't really communicate why Kent's actions are problematic, so I've some sympathy for his not believing that they are.
replies(4): >>45077463 #>>45077573 #>>45077673 #>>45084091 #
4. sevg ◴[] No.45077463[source]
> it doesn't really communicate why Kent's actions are problematic

I agree that the kernel community can be a hostile environment.

Though I’d argue that people _have_ tried to explain things to Kent, multiple times. At least a few have been calm, respectful attempts.

Sadly, Kent responds to everything in an email except the key part that is being pointed out to him (usually his behavior). Or deflects by going on the attack. And generally refuses to apologise.

replies(2): >>45077645 #>>45077786 #
5. bornfreddy ◴[] No.45077492[source]
Being an outsider to this whole scene, the whole thread reads very differently to me.

Kent seems very patient in explaining his position (and frustrations arising from other people introducing bugs to his code) and the kernel & debian folks are performing a smearing campaign instead of replying to what I see are genuine problems in the process. As an example, the quotes that are referenced by user paravoid are, imho, taken out of context (judging by reading the provided links).

There probably is a lot more history to it, but judging from that thread it's not Kent who looks like a bad guy.

replies(4): >>45077710 #>>45077865 #>>45078165 #>>45086496 #
6. yxhuvud ◴[] No.45077573[source]
I've seen plenty of times where the problems has been explained to Kent. But he just don't give a shit about the problems of people that isn't himself or that doesn't use his file system experiences.
replies(1): >>45080324 #
7. ajb ◴[] No.45077645{3}[source]
Definitely not saying that the problems are all on one side here. Agreed that going on the attack was bad (as well as dumb).

I just think that while, yes, the kernel folks have tried to explain, they didn't explain well. The "why" of it is a people thing. Linus needs to be able to trust that people he's delegated some authority will respect its limits. The maintainers need to be able to trust that each other maintainer will respect the area that they have been delegated authority over. I think that Kent genuinely doesn't get this.

8. arp242 ◴[] No.45077673[source]
People have explaining things, at great length, many times. Many of these have been posted to HN before, either as submissions or comments.

Kent just does not listen. Every time the discussion starts from the top. Even if you do agree on some compromise, in a month or two he'll just do the same thing again and all the same arguments start again.

You can't expect people to detail about four or five years of context in every single engagement for the benefit of interested 3rd parties like you or me.

9. johnny22 ◴[] No.45077710[source]
it's waaay simpler than that. Some projects have established rules, and kent doesn't want to follow them. It doesn't matter how nice (or not) he is.
replies(1): >>45080746 #
10. Muromec ◴[] No.45077724[source]
autism is a hell of a social disability sometimes.
replies(1): >>45082894 #
11. philipallstar ◴[] No.45077786{3}[source]
> Sadly, Kent responds to everything in an email except the key part that is being pointed out to him (usually his behavior).

Behaviour sounds like the least important part of code contributions. I smell overpowered, should've-been-a-kindergarten-teacher code of conduct person overreach.

replies(2): >>45078774 #>>45083277 #
12. arp242 ◴[] No.45077865[source]
Kent brings up Debian himself, unprompted.

This is one of the problems: Kent is frequently unable to accept that things don't go his way. He will keep bringing it up again and again and he just grinds people down with it. If you see just one bit of it then it may seem somewhat reasonable, but it's really not because this is the umpteenth time this exact discussion is happening and it's groundhog day once again.

This is a major reason why people burn out on Kent. You can't just have a disagreement/conflict and resolve it. Everything is a discussion with Kent. He can't just shrug and say "well, I think that's a bit silly, but okay, I can work with it, I guess". The options are 1) Kent gets his way, or 2) he will keep pushing it (not infrequently ignoring previous compromises, restarting the discussion from square one). Here too, the Debian people have this entire discussion (again) forced upon them by Kent's comments in a way that's just completely unnecessary and does nothing to resolve anything.

Even as an interested onlooker who is otherwise uninvolved and generally more willing to accept difficult behaviour than most people, I've rather soured on Kent over time.

replies(1): >>45079169 #
13. MadnessASAP ◴[] No.45078165[source]
It may seem like that on the surface, but you should recognise that these sorts of situations seem to follow Kent around.

So either Kent is on a righteous crusade against unreasonable processes within the Kernel, Debian, and every other large software project he interacts with. Or there's something about the way Kent interacts with these projects that causes friction.

I like Bcachefs, I think Kent is a very talented developer, but I'm not going to pretend that he is innocent in all this.

replies(1): >>45080579 #
14. dralley ◴[] No.45078774{4}[source]
No. As someone who likes bcachefs and even literally donates to Kent's patreon, the way he has gone about engaging with the kernel community is not productive. Unfortunately.

CoC isn't even the issue, he constantly breaks kernel development rules relating to the actual code, then starts arguments with everyone up to and including Linus when he gets called out, and aggressively misses the point every time. Then starts the same argument all over again 6 weeks later.

And, like, if you don't like some rules, then you can have that discussion, but submitting patches you know will be rejected and then re-litigating your dislike of the rules is a waste of everyone's time.

replies(1): >>45092001 #
15. koverstreet ◴[] No.45079169{3}[source]
You do realize that data integrity issues are not "live and let live" type things, right?

And there's a real connection to the issue that sparked all this drama in the kernel and the Debian drama: critical system components (the kernel, the filesystem, and others) absolutely need to be able to get bugfixes in a timely manner. That's not optional.

With Debian, we had a package maintainer who decided that unbundling Rust dependencies was more important than getting out updates, and then we couldn't get a bugfix out for mount option handling. This was a non-issue for every other distro with working processes because the bug was fixed in a few days, but a lot of Debian users weren't able to mount in degraded mode and lost access to their filesystems.

In the kernel drama, Linus threw a fit over a repair code to recover from a serious bug and make sure users didn't lose data, and he's repeatedly picked fights over bugfixes (and even called pushing for getting bugfixes out "whining" in the past).

There are a lot of issues that there can be give and take on, but getting fixes out in a timely manner is just part of the baseline set of expectations for any serious project.

replies(1): >>45079488 #
16. arp242 ◴[] No.45079488{4}[source]
Look, I get where you're coming from. It's not unreasonable. I've said this before.

But there are also reasons why things are the way they are, and that is also not unreasonable. And at the end of the day: Linus is the boss. It really does come down to that. He has dozens of other subsystem maintainers to deal with and this is the process that works for him.

Similar stuff applies to Debian. Personally, I deeply dislike Debian's inflexible and outmoded policy and lack of pragmatism. But you know, the policy is the policy, and at some point you just need to accept that and work with it the best you can.

It's okay to make all the arguments you've made. It's okay to make them forcefully (within some limits of reason). It's not okay to keep repeating them again and again until everyone gets tired of it and seemingly just completely fail to listen to what people are sating. This is where you are being unreasonable.

I mean, you *can* do that, I guess, but look at where things are now. No one is happy with this – certainly not you. And it's really not a surprise, I already said this in November last year: "I wouldn't be surprised to see bcachefs removed from the kernel at some point".[1] To be clear: I didn't want that to happen – I think you've done great work with bcachefs and I really want it to succeed every which way. But everyone could see this coming from miles.

[1]: https://news.ycombinator.com/item?id=42225345

replies(2): >>45079527 #>>45081126 #
17. koverstreet ◴[] No.45079527{5}[source]
You have to consider the bigger picture.

XFS has burned through maintainers, citing "upstream burnout". It's not just bcachefs that things are broken for.

And it was burning me out, too. We need a functioning release process, and we haven't had that; instead I've been getting a ton of drama that's boiled over into the bcachefs community, oftentimes completely drowning out all the calmer, more technical conversations that we want.

It's not great. It would have been much better if this could have been worked out. But at this point, cutting ties with the kernel community and shipping as a DKMS module is really the only path forwards.

It's not the end of the world. Same with Debian; we haven't had those issues in any other distros, so eventually we'll get a better package maintainer who can work the process or they'll figure out that their Rust policy actually isn't as smart as they think it is as Rust adoption goes up.

I'm just going to push for doing things right, and if one route or option fails there's always others.

replies(1): >>45080437 #
18. mook ◴[] No.45080172[source]
I think he's too exposed to users reports, because anybody that shows up is in a potential data loss situation. So he's very focused on making everything as bug free as possible, and getting frustrated that people with different focus are not propagating the fixes as fast as possible.

Almost makes me think the distros light-forking it to just change the name (IceWeasel style) so the support requests don't get to him will help… probably not, though, because people will still go there because they want to recover their data.

19. bombcar ◴[] No.45080324{3}[source]
It seems very clear to me that it's almost always a "you can't argue canon law with the Pope" situation - the rules say no new features, and it doesn't matter what the definition of "feature" is if the definition AND the rule come from the same person, Linus.

You can't win a rules-lawyer argument with the rulemaker.

20. simoncion ◴[] No.45080437{6}[source]
> We need a functioning release process...

Yeah, that's in place. If nothing else, the decades of successful releases indicate that the process -at worst- functions. Whether that process fits your process is irrelevant.

> You have to consider the bigger picture.

Right back at you. Buddy, you need to learn how to lose.

21. o11c ◴[] No.45080579{3}[source]
OTOH, the only named projects I've seen are Linux and Debian, which are 2 of the most toxic projects I'm aware of (I'm pretty sure the C++ standards committee beats the two of them combined).

But the problem with comparisons is that even if you're better than nuclear waste being dumped into the aquifer, you still might be enough to light a river on fire.

replies(1): >>45081195 #
22. bornfreddy ◴[] No.45080746{3}[source]
I actually like the idea of the maintainer going out of his way to make sure that my filesystem is safe to use. Even if it goes against the established rules. And I'm saying that as someone who actually likes both Linux and Debian.
replies(1): >>45081951 #
23. nextaccountic ◴[] No.45081126{5}[source]
> But there are also reasons why things are the way they are, and that is also not unreasonable.

It is unreasonable if it leads to users losing data. At this point, the only reasonable thing is to either completely remove support for bcachefs or give timely fixes for critical bugs, there's no middle position that won't willfully lead to users losing their data.

This used to be the default for distributions like Debian some time ago. You only supported foundational software if you were willing to also distribute critical fixes in a timely manner. If not, why bother?

For all other issues, I guess we can accept that things are the way they are.

replies(2): >>45081839 #>>45081934 #
24. JdeBP ◴[] No.45081195{4}[source]
I've been involved in C++ standardization. In my country's national body, it is nothing like what goes on in Linux kernel development, even when there are strong disagreements amongst members.
25. abenga ◴[] No.45081839{6}[source]
> It is unreasonable if it leads to users losing data.

Changing the kernel development process to allow adding new features willy-nilly late in the RC cycle will lead to much worse things than a few people using an experimental file system losing their data in the long term.

The process exists for a reason, and the kernel is a massive project that includes more than just one file system, no matter how special its developers and users believe it is.

replies(1): >>45083357 #
26. rwmj ◴[] No.45081934{6}[source]
Not too familiar with the kernel process for this, but for Linux distros there are ways to respond to critical issues including data corruption and data loss. It's just that you have to follow their processes to do this, such as producing a minimal patch that fixes the problem which is backported into the older code base (and there's a reason for that too: end users don't want churn on their installed systems, they want an install to be stable and predictable). Since distros are how you ultimately get your code into users' hands, it's really their way or the highway. Telling the distros they are wrong isn't going to go well.
replies(1): >>45082218 #
27. rwmj ◴[] No.45081951{4}[source]
It's a strawman to imagine that Debian doesn't have a way to ensure filesystems are safe and to respond to critical bugs that might cause data corruption. It's just that you have to follow their rules to do it. (And broadly the same rules apply to the other big distros as well).
replies(1): >>45091596 #
28. nextaccountic ◴[] No.45082218{7}[source]
For the Debian thing, I'm not sure on the specifics for bcachefs-progs (I'm going by what the author is reporting and some blog posts) but I think the problem with Debian is that they willfully ignore when upstream says "this is only compatible with this library version 2.1.x" and will downgrade or upgrade the library into not supported versions, to match the versions used in other programs already packaged. This kind of thing can introduce subtle, hard to debug bugs. It's a mess and problems are usually reported to upstream, that's a recurrent problem for Rust programs packaged in Debian. Rust absolutely isn't this language where if it compiles, it works, no matter how much people think otherwise.

And this is happening even though it's common for Debian to package the same C library multiple times, like, libfuse2 and libfuse3. This could be done for Rust libraries if they wanted to.

Anyway see the discussion and the relevant article here https://news.ycombinator.com/item?id=41407768 and https://jonathancarter.org/2024/08/29/orphaning-bcachefs-too...

replies(1): >>45082245 #
29. rwmj ◴[] No.45082245{8}[source]
But that's exactly the point here. In the context of a whole distribution, you don't want to update some package to a new version (on a stable branch), because that would affect lots of other packages that depended on that one. It may even be that other packages cannot work with the new updated dependency. Even if they can, end users don't want versions to change greatly (again, along a stable branch). Upstreams should accept this reality and ensure they support the older libraries as far as possible. Or they can deny reality and then we get into this situation.

And carrying multiple versions is problematic too as it causes increased burdens for the downstream maintainers.

I'd argue that libfuse is a bit of a special case since the API between 2 & 3 changed substantially, and not all dependencies have moved to version 3 (or can move, since if you move the v3 then you break on other platforms like BSD and macOS that still only support the v2 API).

Rust and especially Golang are both a massive pile of instability because the developers don't seem to understand that long term stable APIs are a benefit. You have to put in a bit of care and attention rather than always chasing the new thing and bundling everything.

replies(1): >>45083697 #
30. thoroughburro ◴[] No.45082894[source]
Don’t smear all of us with the bad behavior one.
31. jeltz ◴[] No.45083277{4}[source]
No, Kent has generally had a nice tone. The issue is that he has repeatedly violated the rules about code contributions. For example by including new features together with several bug-fixes during rc. That is not a CoC issue, it is not respecting the rules of patch submission and not respecting the time of the kernel maintainers.
32. koverstreet ◴[] No.45083357{7}[source]
There's no need for kernel development process to change. New features go in during RCs all the time, it's always just a risk vs. reward calculation, and I'm more conservative with what I send outside the merge window that a lot of subsystems.

This blowup was entirely unnecessary.

33. rwmj ◴[] No.45083697{9}[source]
BTW here's where I ported nbdfuse from v2 to v3 so you can see the kinds of changes: https://gitlab.com/nbdkit/libnbd/-/commit/c74c7d7f01975e708b...
34. rob_c ◴[] No.45084091[source]
> unable to explain the problem

unfortunately that's either due to lack of investigation by yourself or a bit dishonest.

35. isr ◴[] No.45086496[source]
To be honest, I somewhat agree. I'm sure there's a lot to this that us outsiders don't know (or honestly, for me, couldn't be bothered to know as there are far more important rabbit holes to spend time on).

However, sometimes, a certain detachment can help when looking at what is, in the end, a "cultural disagreement" more suited to an elementary school's playground.

Whenever I see open source spats like this, and then see a Dev harangued and chased from forum to forum by what looks like a coordinated group ("groupies"?), all accusing him/her of rude behaviour, while they keep making attacks on his/her personality, character or temperament...

it leads me to think rather poorly of this "wild west posse".

Anyway, bottom line, Kent is writing open source software to benefit others (and people didn't have qualms about taking his previous bcache & using it to build out storage solutions to make millions), so perhaps he doesn't quite deserve all the abuse and ganging up, no matter whose feathers he ruffled, and how.

36. rcxdude ◴[] No.45091596{5}[source]
And the rules demonstrably create situations where it's easy to introduce bugs or hard to fix them, because they prioritise stability and a consistent set of package versions over the version that the upstream developer has tested. Followed blindly (and without putting in the effort to test to the same level of rigor as upstream), this causes problems, and it's right to point those out. Debian's ways would involve the package maintainer putting in a lot more effort to marry their rules with a package that actually worked, and they were not up for that.

(Debian's rules aren't worthless, it's part of how they can make something that's pretty suitable for 'boring infrastructure' systems because they can keep a system with a known and stable set of behavior up to date with critical security fixes for a long time, but boy do they result in some dumb situations sometimes)

37. philipallstar ◴[] No.45092001{5}[source]
I think it is partly about code of conduct issues[0]. I totally agree that Linus can run whatever release process he likes, and Overbeck should get in line with that. However all of the accompanying sighing at how many times we've had to explain things to him from others is not okay. So what if more discussion is needed or wanted? People doing difficult work might have strong opinions. People doing easy work (e.g. sending code of conduct emails) should not have an equal weight to their opinions, if any at all.

[0] https://lore.kernel.org/lkml/6740fc3aabec0_5eb129497@dwillia...