←back to thread

158 points kenjackson | 1 comments | | HN request time: 0.211s | source
Show context
roblabla ◴[] No.41031699[source]
This is some very poor journalism. The linux issues are so, so very different from the windows BSOD issue.

The redhat kernel panics were caused by a bug in the kernel ebpf implementation, likely a regression introduced by a rhel-specific patch. Blaming crowdstrike for this is stupid (just like blaming microsoft for the crowdstrike bsod is stupid).

For background, I also work on a product using eBPFs, and had kernel updates cause kernel panics in my eBPF probes.

In my case, the panic happened because the kernel decided to change an LSM hook interface, adding a new argument in front of the others. When the probe gets loaded, the kernel doesn’t typecheck the arguments, and so doesn’t realise the probe isn’t compatible with the new kernel. When the probe runs, shit happens and you end up with a kernel panic.

eBPF probes causing kernel panics are almost always indication of a kernel bug, not a bug in the ebpf vendor. There are exceptions of course (such as an ebpf denying access to a resource causing pid1 to crash). But they’re very few.

replies(4): >>41031896 #>>41032164 #>>41032610 #>>41034621 #
josefx ◴[] No.41031896[source]
> likely a regression introduced by a rhel-specific patch. Blaming crowdstrike for this is stupid (just like blaming microsoft for the crowdstrike bsod is stupid).

Yeah, it isn't as if crowdstrike was specifically advertising certified support for RedHat Linux and related products.

https://www.crowdstrike.com/partners/falcon-for-red-hat/

replies(2): >>41032039 #>>41032241 #
roblabla ◴[] No.41032241[source]
Yes, and? They probably do test their software on RHEL.

But how are they supposed to prevent a bug in a newly released kernel update? You can't test your software on future updates that aren't out yet.

If RHEL breaks some core functionality you depend on, in a newly released update, you can't really do much to prevent breakage, even with the best QA in the world. At best, they could have caught it as soon as RHEL published the new kernel... but by then it's already too late, all your currently-deployed probes now have a ticking time bomb, and need to be updated before the RHEL kernel update is applied, lest you kernel panic.

replies(1): >>41032327 #
hsbauauvhabzb ◴[] No.41032327[source]
Maybe by not loading the module into unknown kernels in the first place?

If say you support a distro, you can’t turn around and complain that supporting the newest version is hard, no matter who caused the problem. Plenty of products say ‘this works on $x but it’s not officially supported’.

replies(3): >>41032799 #>>41033155 #>>41034502 #
1. freedomben ◴[] No.41034502[source]
> Maybe by not loading the module into unknown kernels in the first place?

Then you better tell your customers not to `dnf update` until you've had a chance to whitelist the new kernel and ship it in your own stream. Otherwise everyone who updates before you do ends up broken. If a vendor told me that, I would laugh, realize they were serious, thank them for their time but let them know that we will be going a different direction.