CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes

This is some very poor journalism. The linux issues are so, so very different from the windows BSOD issue.

The redhat kernel panics were caused by a bug in the kernel ebpf implementation, likely a regression introduced by a rhel-specific patch. Blaming crowdstrike for this is stupid (just like blaming microsoft for the crowdstrike bsod is stupid).

For background, I also work on a product using eBPFs, and had kernel updates cause kernel panics in my eBPF probes.

In my case, the panic happened because the kernel decided to change an LSM hook interface, adding a new argument in front of the others. When the probe gets loaded, the kernel doesn’t typecheck the arguments, and so doesn’t realise the probe isn’t compatible with the new kernel. When the probe runs, shit happens and you end up with a kernel panic.

eBPF probes causing kernel panics are almost always indication of a kernel bug, not a bug in the ebpf vendor. There are exceptions of course (such as an ebpf denying access to a resource causing pid1 to crash). But they’re very few.

> just like blaming microsoft for the crowdstrike bsod is stupid

Wait, how is this stupid? Unless I'm missing something, wasn't the patch part of a Microsoft payload that included an update to Crowdstrike? Surely Crowdstrike is culpable, but that doesn't completely absolve Microsoft of any responsibility, as its their payload.

Microsoft should revoke the CrowdStrike driver signature and should do an internal check as to why CrowdStrike's driver was approved when it can execute arbitrary code on the kernel level without any checks. If your "driver" requires this feature MS should require CrowdStrike to submit the entire source and they should have to pay MS to do a review of the code.

What is the point of driver signing if a vendor can basically build in a back door and Microsoft doesn't validate that this back door is at least somewhat reasonable

Do you think Microsoft customers using CrowdStrike would then be happier, being unable to run the software at all, due to an action Microsoft took?

Backdoors of all kinds can be installed to most any operating system without vendor co-operation. That is the nature of general-purpose operating systems.

At this point… yes.

It would be one thing Microsoft could do to focus 100% of the attention/blame away from Windows and onto CloudStrike. And customers will want their pound of flesh from somewhere.

Really, this should serve as a wake up call w/in Microsoft to start to harden the kernel against such vulnerabilities.

Was the crash the fault of Windows? No. But did a Windows design decision make this possible? yes.

I’m sure the design decision made sense at the time (at least business sense). Keeping the kernel more open for others to add drivers to makes it easier to write/add drivers, but makes the system more vulnerable. This a good opportunity within Microsoft to get support for changing that.

Ultimately this would have been almost a non-issue if there had been better deployment strategies in place for also the data file updates.

If by changing the system you mean adding some kind of in-kernel isolation to it, then I don't think it would be worth the effort to make that kind of major change to the way operating systems work just to give arguably a minor risk reduction to systems—in particular if CrowdStrike and other vendors take some learnings from this event.

Microsoft might improve their system rollback mechanism to also include files that are not strictly integrated to the system, merely used by the parts that are (the channel files loaded by the driver).

Actually I think we can just be happy that the incident was a mistake, not an attack. Had this kind of "first ever" situation been an attack, it could be extremely difficult to recover from it. I wonder how well EDRs deal with "attacks from within"..

CrowdStrike pulled off the update within 1.5 hours. I wonder if they actually use Falcon themselves? But then somehow missed the problem? Doesn't seem like they eat their own dog food :). (Or at least their own channel files.)

There's a simple thing microsoft could do to avoid this, that doesn't require anything too crazy. EDRs work in kernel-land because that's the only place you can place yourself to block certain things, like process creation, driver loading, etc...

macOS has a userland API for this, called EndpointSecurity, which allows doing all the things an EDR needs, without ever touching kernelland. Microsoft could introduce a similar API, and EDRs would no longer need a driver.