Linux 4.5-rc5: efivarfs fixed to avoid “rm -rf /” bricking UEFI

> but the developer's thought that it was working as intended

Really? Is that evidenced by Lennart's response to this, which stated "The ability to hose a system is certainly reason enought to make sure it's well protected and only writable to root."[1]? I think it implies the opposite.

1: https://github.com/systemd/systemd/issues/2402

The firmware author(s) should not brick their hardware when EFI variables are deleted. That is an invalid behavior.

Root is certainly capable, traditionally, of completely hosing all levels of the software stack. Usually it's the OS's job to protect even root from being able to hose hardware/firmware, especially by easy-to-make mistake.

Agreed, but that's almost a non-sequitur. The systemd developers are not the firmware developers. Additionally, while preferably we would have better firmware that couldn't be bricked in this way, we live in a world where this is often not the case, so the problem does need to be mitigated in some manner.

Yes, which is what we have here. My point is that I don't think there was ever any indication that the systemd developers thought there wasn't a problem to be fixed. That the fix was pushed farther up-stack so it was more comprehensive is not a bad thing in my eyes.

Put another way, systemd mounting efivarfs read-only doesn't necessarily prevent you from bricking your machine by deleting files within that file system, it protects only those using systemd. The problem stems from the nature of efivarfs, and that it was implemented as a file system and not some other interface. Making it default to something more sane is a better (but by no means the best) solution, and that ideally would happen within the source, which is the kernel.

I was going from memory, and didn't reread Lennart's responses in the thread. But looking at it again now, I still think it's an accurate summary. Lennart's line after you quoted is "But beyond that: root can do anything really."

I read this as Lennart saying that when root issues an 'rm' in efivarfs, the variable should be removed even if this renders the motherboard unusable without physical repairs. What's your interpretation?

I've edited to fix my terrible punctuation, and to make it clear that 'it' refers to 'systemd', and to add a link to MJG's response on Twitter. I can edit further if you have a way to make it clearer.

It shouldn't be mounted rw, and only exposed at root. It shouldn't be mounted by default. It shouldn't even be a bloody filesystem. EFI variables have so much metadata and other information attached to them that trying to wedge them into the filesystem is just asking for problems like this. Were the maintainers smart, they'd dump the file system all together, because it's a nonsensical way of presenting this data.

You're missing the end of that quote:

> But beyond that: root can do anything really.

... so running "rm -rf /" as root should brick your motherboard because it's the responsibility of the motherboard manufacturer to protect against this. That's all fine and dandy in an idealized world, but in the "real world" there are going to be motherboard manufacturers that play fast and loose with these things.

The amount of software that requires write access to EFI variables (as root) is pretty small, so leaving it open all of the time in an environment where the firmware writer could have bugs that cause this seems like it's not necessarily the best course of action.

I was saying the same thing a while ago but nobody takes it seriously.

> It shouldn't be mounted rw

It should be mounted rw because existing userspace expects it to be rw.

> It shouldn't be mounted by default.

It should be mounted by default because it's information that's relevant to various pieces of userspace.

> It shouldn't even be a bloody filesystem.

With hindsight, it should absolutely not have been a filesystem. There's very little metadata associated with EFI variables and the convenience of exposing this in a way that can be handled using read and write is huge, but real-world firmware turns out to be fragile enough that this was a mistake. But, in the absence of a time machine, there's very little I can do to fix that now.

It's firmware. It's supposed to brick your system if you ruin it. Firmware is basically hardware. It's not a multi-user protected-mode operating system.

It's the responsibility of the OS to make sure it doesn't ruin firmware.

This wasn't removing firmware as in the actual boot code for the motherboard. It was just clearing the list of what drives to attempt to load an OS from. That list is explicitly intended to be accessible to and modifiable by operating systems, to provide a better user experience than the old method of the user having to manually change BIOS settings.

The standard practice for PCs has always been that firmware configuration settings can be cleared (through a jumper or by pulling the battery) to reset the system to its factory state, forcing it to fall back to its conservative and safe defaults. Some systems have apparently forgotten to have defaults. Their firmware is already broken and afflicted with a major bug even if you avoid triggering it in this particular manner.

It shouldn't be possible to brick hardware by running 'rm'.

I don't disagree with this, but I still think it's irresponsible of systemd to claim it's a kernel problem but not take steps to help people not brick their systems before it's fixed upstream.

Start pushing for its deprecation. Devfs existed before it was replaced by udev. Keeping a broken system around just makes it harder to fix it. Interfaces on other systems, like the BSDs' sysctl show that you don't need to make everything a pseudofilesystem to make variables possible to change. If someone really really wants their efi variables as a filesystem, they could always write a fuse driver or the like.

No, you are missing the context of the earlier paragraph, which says "Well, there are tools that actually want to write it." so it needs to be accessible in some way. Making it truly read-only is not really feasible, and goes against the Unix philosophy of "root can do anything." That's not to say that we shouldn't make it harder for root to do some things, such as brick your system. In the end, the fix still doesn't prevent root from bricking a system in the case the firmware is badly coded, it just makes it a bit harder. Mounting read-only by default or defaulting all files to immutable until changed by chattr are not hard hurdles to overcome for root, but they the do make it harder than it would be without them, which in this case is a good thing.

> ... so running "rm -rf /" as root should brick your motherboard because it's the responsibility of the motherboard manufacturer to protect against this.

You are assuming "only writable by root" means "root can write without any restrictions" which is a fairly uncharitable reading, and requires assumptions about his intent which are not evident.

I could make a statement such as "cars in the united states can only be legally driven by people of an appropriate age" and you could assume I meant that's all that needs to apply and start calling out my statement as wrong, or you could assume I was aware of the additional requirement of a driver's license, or you could ask me to clarify my point of view. I just don't believe the first option is conducive to useful discussion, nor do I think it's appropriate to use assumed information in a negative way towards a third party.

Edit: s/disparage/use assumed information in a negative way/ for lack of better phrasing coming to mind. The statement wasn't really disparaging, just an uncharitable interpretation, so I don't want to overstate that.

> It should be mounted rw because existing userspace expects it to be rw.

Can you explain this a bit? I'm not familiar with the particulars of firmware, but I'm having a hard time imagining why any userspace program would expect a firmware partition to be writable. Even if there are any, certainly I'd have a hard time believing that they would need it to be mounted and writable all the time.

We should also live in a world without kernel panics and BSODs, too.

It's not a firmware partition, it's an interface to the nvram variables provided by the firmware. There's a bunch of reasons why these should be writable - you need to be able to modify some of them to do things like reboot into the firmware settings UI, they're used for secure boot key management, you want to be able to choose which OS to boot into on the next reboot or which network interface to PXE off, that kind of thing.

As for why userspace couldn't remount it itself - it could. It doesn't. Changing the behaviour of systemd without changing the behaviour of the rest of userspace would result in userspace being broken, and making that kind of incompatible change is annoying - especially when fixing it in the kernel allows us to avoid that breakage.

What, like his second comment (which was just a few minutes after his first, IIRC), which in it's entirely is '(note that you can remount it readonly at boot, simply by adding an entry for it into /etc/fstab, that is marked "ro")'? Keep in mind we've had a single month since this bug was submitted. Systemd had a release 11 days ago according to their repo[1], but if that wasn't expected to be picked up and used by distros (which it might not, depending on the distro and what changes have been implemented since the distro shipped/updated systemd), then the appropriate places to fix this are in the kernel and/or as a patch to the distro package for systemd.

Really, I think the two appropriate places for this fix are the kernel, and if that's not expected to be rolled out soon, as a patch to systemd to mount it read-only by the distro shipping systemd. Systemd shipping the fix would only really help the small group of people that install systemd from source during the short window from 11 days ago until the next kernel is available (or that choose to run an older kernel), and makes the whole efivarfs situation a bit more confusing by leaving it read-only and immutable for the future.

1: https://github.com/systemd/systemd/releases/tag/v229

My interpretation, stemming from his statement that it is a problem to brick machines and his statement that some programs need write access, is that something needs to be put in place, but we can't unilaterally restrict root (which would be very un-Unixy to do). I think he was fairly ambigous in details on how to fix it (possibly because he wasn't sure the best path to take), and included with some fairly blanket statements about policy/belief that allowed people to interpret his statements however they were inclined. Unfortunately due to the polarizing effect of systemd (and Lennart's prior projects, and possibly Lennart himself), there are plenty of people inclined to believe he doesn't care.

> Can you explain this a bit? I'm not familiar with the particulars of firmware, but I'm having a hard time imagining why any userspace program would expect a firmware partition to be writable.

Best example of why you need to have it RW: if uefi is in fast boot, the only way to actually enter uefi is to boot an OS and have the OS set the uefi variables so that it goes into the firmware screen on next restart. This is (on Linux) done by changing something in that virtual filesystem.

> certainly I'd have a hard time believing that they would need it to be mounted and writable all the time.

You could probably remount it, but that also means that you all the sudden have concurrency problems with that operation. So not really ideal.

The kernel should mount it read-only. If root wishes to delete or modify the contents, root can remount it read-write.

> goes against the Unix philosophy of "root can do anything."

I don't think that's a useful perspective here. This is not a feature - nothing should ever want to permanently brick a motherboard, there's no use case for that. There's no benefit to allowing root to do this. The OS is supposed to abstract the hardware in a way that it can be safely operated.

I think you are mistaken about some of the details here. We are talking about root being able to write to efivarfs, which is needed for some valid reasons. Unfortunately, there are firmware implementations out there that don't handle some variables being overridden/wiped, and it bricks the system. So, the problem is bad firmware, exacerbated by the choice to represent EFI variable as a filesystem and what happens when you do a "rm -rf /" (in the simplest example). So, in my eyes, root should be able to write to efivarfs when needed, but we should mitigate this problem in some way so it's hard to accidentally brick your system by making it a little harder (but not impossible) for root to write to this filesystem. It's not possible to allow root to write to EFI while still ensuring that it can't brick the machine, because the responsibility for that capability lies with the firmware developers.

Right, but it should be possible to brick hardware through firmware updates.

Again, this is the OS's fault.

For this particular example, wouldn't this be best solved with a separate runlevel for "reboot into UEFI"? Stop all daemons, remount all normal filesystems read-only, remount UEFI RW, write variable and reboot.

Not everything is protected, like if the user is using chmod -R to change permissions to 777 and makes a typo for the main directory / instead of ./ or ~/ and then the whole filesystem is 777 and things stop running due to wrong permissions.

Right. That's the software stack. You may have to reinstall your OS at that point, but your motherboard is definitely not bricked.

Why is it the OS's fault if the firmware can't reset itself to factory defaults? You're not modifying the firmware image, you're modifying a scratch area exposed by the firmware for the express purpose of the OS (ab)using it in any way it deems necessary.

The kernel isn't in charge of mount options, that's the job of userspace (generally an init system). In any case, you would end up needing a way to mount it RW to accommodate legit uses of the non-broken EFI vars, at which point you're back where you started. This fix addresses it by having the kernel identify and protect only broken (or at least, non-standard) EFI vars.

It happens rarely, but actually I totally agree with Lennart on this one. Maybe not for the very same ultimate reasons, but nevertheless, I agree.

Being able to brick hardware through a very oftenly used action (unliking a filesystem entry) throws us back into the times where one could damage display devices beyone repair by feeding them scan frequencies outside their operational range or by destroying hard disks by smashing the heads into a parking position outside of the mechanical range. We left those days behind us some 20 years ago: Devices got smart enough to detect potentially dangerous inputs and execute failsafe behaviour. It's just reasonable to expect this from system firmware.

When talking about (U)EFI variables we're not talking about firmware updates, which are kind of a special action (and even for firmware updates its unacceptible that a corrupted update bricks a system¹). Manipulating (U)EFI variables is considered a perfectly normal day-to-day operation and the OS should not have to care about sanity checks and validity at boot time. (U)EFI is the owner and interpreter of these variables, so it is absolutely reasonable to expect the firmware to have safeguards and failsafe values in place.

IMHO (U)EFI is a big mess, a bloated mishap of system boostrap loader. And I'm totally against trying to workaround all the br0kenness in higher levels. The more and often systems brick due to the very fundamentals of (U)EFI being so misguided, the sooner we'll move on to something that's not verengineered.

----

¹: Just to make the point: When we developed the bootstrap loader for our swept laser product we implemented several safeguards to make it unbrickable. It's perfectly fine to cut the power or reset the device in the middle of a firmware upgrade. It safely recovers from that. Heck, the firmware in flash memory could become damaged by cosmic radiation, the bootloader would detect it and reinstall it from a backup copy in secondary storage.

Aside: I misread "swept laser" as "sweet laser" and was really bummed you didn't include a link to the Kick Starter for whatever project that would have been.

I completely disagree. I looked at the patch above. I personally don't like. mounting efivars rw is akin to mounting boot rw by default.

A sane linux distro will mount it ro and switch to rw whenever need. Defaulting to rw efivars is, excuse the language stupid.

I've done a fair share of efi debugging even removing some of the variables that the kernel will now protect you from breaking.

If the issue is that users should be able to remount efivars as rw whenever needed then that should be addressed, not prevent you from doing stuff to it because there is a rogue init system doing crazy stuff.

EDIT: BTW, i don't think systemd does anything besides write to the various Boot* variables, but I may be wrong. I don't see why that can't be addressed with a remount. If you replace the boot.efi you still have to remount the efi partition anyway.

While Matthew may be right that there is an issue that needs to be addressed, but in one of his tweets he basically says the kernel should fix it because tooling isn't and bioses suck. Well, maybe tooling should be forced to fix it.

or from the issue:

    Matthew-Jemielity commented 24 days ago
    What needs efivars mounted at all anyway? So far I've seen:

    grub
    systemctl --firmware-setup reboot
    efibootmgr
    Since those likely need superuser, couldn't they handle (un)mounting it themselves?
    
    @annejan
    annejan commented 23 days ago
    As long as distribution that are aimed at consumers remount it ro and on updating kernels wrap grub with remount this is a complete non-issue.

The EFI variables are just that. Variables not an entire firmware. I think this is where you confusion is. What is happening here is that these variables are presented to the user as a filesystem under /sys. If you run 'rm -rf /' (or even 'rm -rf /sys'), you will start "deleting" these variables (which I assume either sets them to a null-equivalent value, or somehow removes them entirely from wherever they are stored). It sounds like:

  rm /sys/efivar/MyVariable

is roughly equivalent to:

  var obj = {MyVariable: 1};
  obj.MyVariable = null;

or:

  var obj = {MyVariable: 1};
  delete obj.MyVariable;

This bricks the firmware, because some of these variables end up being required. It's on the firmware maker / motherboard manufacturer to make sure that there is a way to recover from this rather than having the firmware fail to startup because some variables are missing.

My understanding is that firmware made to spec would not brick over the the EFI variables getting wiped. The motherboards that are encountering these issues are running firmware that is cutting corners. Unfortunately cutting corners and ignore specs is nothing new for hardware manufacturers who are more concerned with the manufacture of the physical devices and usually put only the minimum amount of effort into getting any sort of software components (firmware, OS device drivers, etc) running.

My understanding is that the amount of software that wants/needs to write to efivarfs is fairly small[1] compared to the amount of software that is normally run with root privileges.

The fact is that 'rm -rf /' is a common mistake, both at the command line (i.e. manually typing) and in scripts that don't adequately protect against things like missing variables. E.g.:

  MY_DIR=
  rm -rf "$MY_DIR/"

The fact that this could brick a system is a big deal. Pushing blame around doesn't do anybody any good.

[1] I realize that boot loaders like grub are everywhere and probably need to write to efivarfs, but that's still a data point of 1. Would it be that difficult for grub and it's related scripts to upgrade to remount the filesystem readwrite when it needs to perform an operation? I'm sure it's only been a couple of years since efivarfs functionality was even added to grub.

> In any case, you would end up needing a way to mount it RW to accommodate legit uses of the non-broken EFI vars, at which point you're back where you started.

Not necessarily. Leaving it "wide open" for anything to accidentally write to it all of the time vs. just mounting it readwrite when you actually need to write to it are two different risk profiles.

I can see the point that you shouldn't make otherwise-undesirable OS changes to accommodate incompetent hardware designers. The thing is, I'm not convinced that it this is undesirable: mounting efivars read-only by default seems like a completely reasonable thing to do, even absent this particular hardware issue. BIOS setting are important and rarely need to be changed, and anything that doesn't have the privileges to unlock efivars probably shouldn't be messing with it anyway.

I understand and agree with all of this, I'm just not sure how it follows from what I wrote. The problem has a fix scheduled (what this submission is about), and in a fairly short period of time (~30 days for a fairly esoteric[1] bug).

To my mind there are two places to fix this, in the kernel for a real mitigation technique that helps "solve" the problem, and in the distros for quick fixes and hacks, or backports of the kernel fix, as necessary. Systemd pushing a fix of their own 1) only affects distros using systemd, while this problem affects all recent distros that use efivarfs), 2) probably won't get picked up by distros immediately excapt as a backported fix anyway, as I doubt most of them push new versions of something as integral as systemd every time a new version is out, at least not with a lot of testing, and 3) would not have been a good fix, and would have required the utilities that still needed access to actually remount a filesystem.

1: Triggering this problem is not as easy as what you (and I) wrote in most instances, as / has special consideration in rm, and generally requires the "--no-preserve-root" flag.

Bingo! While preventing a user from accidentally hosing their system due to a bad firmware implementation this still does nothing to address potential malware, it's good to have a patch that helps but I think any blame directed at systemd or the kernel devs really needs to be redirected at poor firmware engineering efforts. Mounting efivars as ro is a bandaid, this patch is a bandaid - the real damage is yet to be done!

> I personally don't like. mounting efivars rw is akin to mounting boot rw by default.

No, it's not the same. Mounting `/boot` rw by default does not put your system in the danger of getting damaged beyond repair. If you hose the boot partition you can always start a recovery system (live Linux or similar) to repair the damage.f

But if deleting efivars renders a system inoperable on a firmware level you're essentially SOL, save for rewriting the contents of the system firmware flash using an external programmer and a clean image. That is an absolutely inacceptable situation. The year is 2016 and hosing a firmware by writing malformed values into the firmware API is, simply put, a software vulnerability that allows to permanently DoS a system. As such this is a security issue that must be fixed at where the security issue happens. And in case of efivars the issue is that certain input is not properly validated and/or sanitized. If a system firmware can not properly start with certain variables being unset or removed or set to invalid valued, its should be a implementation requirement to validate input on such variables before executing the change.

> Defaulting to rw efivars is, excuse the language stupid.

It probably it. But it's not the responsibility of the OS to sanitize values that are not intended for being used by the OS. efivars are intended to be used by (U)EFI and hence it's the (U)EFI implementation's task to properly sanitize access to them.

Essentially we're talking Bobby Tables here, just with a different API.

> I totally agree with Lennart on this one.

There's a certain rich irony that Lennart is being flogged for following the Unix tradition (root can do anything, including blow up their monitor with bad X configs); that his detractors are suggesting userland tools manage their hardware (normally the job of the kernel in a Unix system); that systemd ought to be expanded to manage hardware (after years of complaining it's too big), presumably by adding whitelisting capabilities and a database of known-good/known-bad UEFI implementations.

I guess at least it demonstrates how utterly unhinged some people become when his name is attached to anything.

> mounting efivars read-only by default seems like a completely reasonable thing to do, even absent this particular hardware issue.

It's perfectly reasonable to ro mount efivars. But doing so must not be the workaround to fix an security issue (and yes, this is a security issue) in (certain implementations of)(U)EFI. Just to make clear why this is an security issue: Security rests of three pillars:

- availability - confidentiality - authenticity

Rendering a system unusable (DoS-ing it) it an attack on availability. And speaking of security, if the goal is sabotage and causing large financial damages, then being able to permanently brick a system in case of a privilege escalation (there's nothing stopping UID=0 from remounting rw efivars) is pretty bad; and no, the implemented fixes in the efivars kernel code don't help, because an attacker can still mount a custom kernel module which will talk to the respective efivars code directly circumventing sanity checks (or directly talk to (U)EFI without using the efivars code).

I remember the days when a single virus destroyed practically every motherboard for my local government. They had to throw away every machine and replaced it, costing the Swedish government several millions in early 2000 (https://en.wikipedia.org/wiki/CIH_%28computer_virus%29), and according wikipedia caused globally something around $1 billion US dollars in commercial damages.

A virus should not be able to destroy the system BIOS. The problem with efivars is illuminating a vulnerability, not a feature for "doing stuff". I would expect to see this used in the wild if left unfixed, especially the next time we hear about a remote vulnerability that permit arbitrary memory execution.

> (U)EFI implementation's task to properly sanitize access to them.

If the implementation is bad (which it is), that doesn't excuse reckless userland software that refuses to acknowledge flaws in the firmware.

There's a layer between userland and firmware where those things should be handled, and indeed everyone agreed that it should be handled in the kernel

Saying that Lennart is wrong has become a rather popular sport, let's not go overboard to say it even when he's right by all accounts.

Your approach still bricks motherboards if somebody forgets to remount ro after having remounted rw.

The kernel fix doesn't have such drawback.

Possibly, but what happens if a faulty script does a `rm` too much in that short time window?

The kernel fix prevents that, using mount flags alone only restrict the vulnerability but it doesn't make it go away.

> I guess at least it demonstrates how utterly unhinged some people become when his name is attached to anything.

He earned it with his hard work with the Linux community.

By propelling linux process supervision into the 21th century, change is scary

No. That one earned him adoption of his init.

Actually Lennart working for RedHat is what gave systemd serious traction. If it came to 21st century software, there are numerous modern init systems around (dependency driven, fast) that largely predate systemd. If any of those systems had been widely adopted by now it would be probably far ahead of systemd in feature set.

OpenRC is what got closest to be the replacement SysV-Init, just by its qualities and market share. But other, nicer systems exist, but they never found wide adoption because the main distributions didn't let go of SysV.

Now Lennart claims all the bragging rights, but people who used modern init systems before it was cool know better (there are a lot of Lennart opponents among those, BTW).