The Therac-25 Incident (2021)

The most deadly bug in history. If you know any other deadly bug, please share! I love these stories!

The MCAS related bugs @ Boeing led to 300+ deaths, so it's probably a contender.

The 737 Max MCAS is arguably a bug. That killed 346 people.

Not a "bug" per se, but texting while driving kills ~400 people per year in the US. It's a bug at some level of granularity.

To be tongue in cheek a bit, buggy JIRA latency has probably wasted 10,000 human years. Those are many whole human lives if you count them up.

Was that a bug or a failure to inform pilots about a new system?

Probably many rather than a single bug, but the botched London Ambulance dispatch software from the 90s, is probably one of the most deadly software issues of all time, although there aren't any estimates I know of that try to quantify the number of lives lost as a result.

http://www0.cs.ucl.ac.uk/staff/a.finkelstein/papers/lascase....

Both - and really MCAS was fine but the issue was the metering systems (Pitot tubes) and the handling of conflicting data. That part of the puzzle was definitely a bug in the logic/software.

In the same vein one could argue that Therac-25 was not actually a software bug but a hardware problem. Interlocks, that could have prevented the accidents and that where present in earlier Therac models, were missing. The software was written with those interlocks in mind. Greedy management/hardware engineers skipped them for the -25 version.

It's almost never just software. It's almost never just one cause.

Several people killed themselves over this: https://www.wikipedia.org/wiki/British_Post_Office_scandal

https://www.theguardian.com/uk-news/2024/jan/09/how-the-post...

One member of the development team, David McDonnell, who had worked on the Epos system side of the project, told the inquiry that “of eight [people] in the development team, two were very good, another two were mediocre but we could work with them, and then there were probably three or four who just weren’t up to it and weren’t capable of producing professional code”.

What sort of bugs resulted?

As early as 2001, McDonnell’s team had found “hundreds” of bugs. A full list has never been produced, but successive vindications of post office operators have revealed the sort of problems that arose. One, named the “Dalmellington Bug”, after the village in Scotland where a post office operator first fell prey to it, would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for.

Another bug, called the Callendar Square bug – again named after the first branch found to have been affected by it – created duplicate transactions due to an error in the database underpinning the system: despite being clear duplicates, the post office operator was again held responsible for the errors.

Just to point it out even clearer - there's almost never a root cause.

Remember the Airbus that crashed in the middle of the Atlantic because one of the pilots kept pulling on his yoke, and the computer decided to average his input with normal input from the other pilot?

Conflict resolution in redundant systems seems to be one of the weakest spots in modern aircraft software.

More heads should have rolled over this in my opinion, absolutely despicable that they cheerfully threw innocent people in prison rather than admit their software was a heap of crap. It makes me so angry this injustice was allowed to prevail for so long because nobody cared about the people being mistreated and tarred as thieves as long as they were 'little people' of no consequence, while senior management gleefully covered themselves in criminality to cover for their own uselessness.

It's an archetypal example of 'one law for the connected, another law for the proles'.

I would say plenty of both. They obviously had to inform the pilots, but the way the system didn't reset permanently after 2-3 (whatever) sessions of "oh, the pilot trimmed manually, after 10 seconds we keep doing the same thing" was a major major logic blunder. Failure all across the board, if only from the perspective of end-to-end / integration testing if nothing else.

Worryingly, e2e / full integration testing was also the main cause of other Boeing blunders, like the Starliner capsule.

Not even close. Israel apparently has AI bombing target intel & selection systems called Gospel and Lavender - https://www.theguardian.com/world/2024/apr/03/israel-gaza-ai.... Claims are these systems have a selectivity of 90% per bombing, and they were willing to bomb up to 20 civilians per person classified by the system as a Hamas member. So assuming that is true, 90% of the time, they kill one Hamas member, and up to 20 innocents. 10% of the time, they kill up to 21 innocents and no Hamas members.

Killing 20 innocents and one Hamas member is not a bug - it is callous, but that's a policy decision and the software working as intended. But when it is a false positive (10% of the time), due to inadequate / outdated data and inadequate models, that could reasonably classified as a bug - so all 21 deaths for each of those bombings would count as deaths caused by a bug. Apparently (at least earlier versions) of Gospel were trained on positive examples that mean someone is a member of Hamas, but not on negative examples; other problems could be due to, for example, insufficient data, and interpolation outside the valid range (e.g. using pre-war data about, e.g. how quickly cell phones are traded, or people movements, when behaviour is different post-war).

I'd therefore estimate that deaths due to classification errors from those systems is likely in the thousands (out of the 60k+ Palestinian deaths in the conflict). Therac-25's bugs caused 6 deaths for comparison.

Some Google Pixel phones couldn't dial emergency number (still can't?). I don't know if there were any deadly consequences of that.

https://www.androidauthority.com/psa-google-pixel-911-emerge...

> To be tongue in cheek a bit, buggy JIRA latency has probably wasted 10,000 human years. Those are many whole human lives if you count them up.

These kind of calculations always make me wonder...say someone wasted one minute of everybody's life, is the cost ~250 lives? One minute? Somewhere in between?

That wasn't a bug.

They deliberately designed it to only look at one of the Pitot tubes, because if they had designed it to look at both, then they would have had to implement a warning message for conflicting data.

And if they had implemented a warning message, they would have had to tell the pilots about the new system, and train them how to deal with it.

It wasn't a mistake in logic either. This design went through their internal safety certification, and passed.

As far as I'm aware, MCAS functioned exactly as designed, zero bugs. It's just that the design was very bad.

Not a bug. A non airworthy plane they tried to patch up with software.

The plane was perfectly airworthy without MCAS, that was never the issue. The issue was it handled differently enough at high angles of attack to the 737NG that pilots would've needed additional training or possibly a new type rating without MCAS changing the trim in this situation. The competition (Airbus NEO family) did not need this kind of new training for existing pilots, so airlines being required to do this for new Boeing but not Airbus planes would've been a huge commercial disadvantage.

[edit as I can't reply to the child comment]: The FAA and EASA both looked into the stall characteristics afterwards and concluded that the plane was stable enough to be certified without MCAS and while it did have more of a tenancy to pitch up at high angles of attack it was still an acceptable amount.

I may have understood wrong but thought is possible to get into an unrecoverable stall?

There was a news story from Norway last year where a car allegedly accelerated by itself, causing the car to fall off the second floor of a parking garage and kill the driver.

Air France 447: https://en.m.wikipedia.org/wiki/Air_France_Flight_447

Inputs were averaged, but supposedly there’s at least a warning: Confused, Bonin exclaimed, "I don't have control of the airplane any more now", and two seconds later, "I don't have control of the airplane at all!"[42] Robert responded to this by saying, "controls to the left", and took over control of the aircraft.[84][44] He pushed his side-stick forward to lower the nose and recover from the stall; however, Bonin was still pulling his side-stick back. The inputs cancelled each other out and triggered an audible "dual input" warning.

There are plenty of "car allegedly accelerated by itself" incidents, and usually the root cause is the driver mistakenly pressing the accelerator pedal when they think they're pressing the brake pedal. And then swearing blind afterwards that they were braking as hard as they possibly could but the car kept surging forwards.

It wasn't pitot tubes that had the hardware problem, it was the angle of attack sensor. The software was poorly designed to believe the input from just one fallible angle of attack sensor.

In Dhahran, Saudi Arabia, on February 25, 1991, a Patriot missile failed to intercept an Iraqi Scud causing the death of 28 American soldiers.

The patriot missile system used floating point for time, so as uptime extended the clock became more and more granular, eventually to the point where time skipped so far that the range gate was tripped.

The fix was being deployed earlier that year but this unit hadn't been updated yet.

https://www.cs.unc.edu/~smp/COMP205/LECTURES/ERROR/lec23/nod...

Time and time again the introduction of electronic throttle control has spiked the number of reported unintended acceleration incidents.

There's a chart here that shows it clearly for Toyota's rollout:

https://www.embedded.com/unintended-acceleration-and-other-e...

That chart is not clear.

The Camry, the Solara, and the RAV4 are all the exact same engine hardware and software, at nearly all times. Especially the Solara, which is just a Camry with a shinier body. It uses an identical engine, throttle body, and ECU flash and is even considered a "Camry Solara".

That Camry "Unintended accelerations" jumped while Solara didn't means that it isn't the hardware. Instead, they all started at the same time, about 2002. Similarly, the Scion TC is also literally the exact same hardware, software, and throttle body as a Camry. The entire Scion line is just Toyota software and hardware in a different body shell.

Indeed, if you look at the Corolla, the jump in unintended acceleration cases start with mechanical throttle bodies still the norm, and do not change with the switch to electronic throttle control.

IMO this graphic handily shows how the media affects average people. The media went on a large blitz about how Toyota was unsafe now that they had electronic throttle bodies, and so owners of those cars complained, but the average consumer does not realize that the Solara, Camry, and Scion TC are all identical vehicles under the body shell and don't realize that they "should" also be complaining about those cars if the problem was actually caused by the electronic throttle body code or design.

Important note: People who report unintended acceleration events like this almost always say that the brakes didn't stop it. That seems.... hard to believe. The brakes on all Toyotas are fully hydraulic and cannot fail electronically. All toyota vehicles in that chart (maybe excluding some Tacomas and the top line Lexus model) have brake capability far exceeding their engine power. A V6 Camry can be at wide open throttle but hard application of the brakes would still overpower that engine with no problem.

Interestingly the NHTSA's opinion is that the Toyota models in the graphic ARE defective: They allow faulty or improperly installed weather mats to interfere with the pedals. Toyota also believed this take, as they kept their handling of fixing this defect off the books, and changed the pedal positioning in their newer models. They eventually fined Toyota over a billion dollars for their handling of this situation, and claimed there was another "sticky pedals" problem that they were covering up.

> The brakes on all Toyotas are fully hydraulic

No, most are hydraulic with vacuum boost.

If you aren't expecting it, the increased force required to drive the brakes hydraulically from the pedal without boost assist is significant and can be surprising. I assume most folks haven't had an engine fail going downhill, but for a large pickup I was standing on the brake pedal and had to push my leg down by pulling on the steering wheel to stop it.

> That seems.... hard to believe

Anyhow, the vacuum booster is driven from the engine airflow. At wide open throttle the vacuum available to the booster is minimal because the restriction is as open as it can be.

You can test how it feels by rolling at a medium speed in a parking lot, shifting into neutral, and killing the engine. The vacuum reservoir may provide you one or two brake pumps and then you're on your own.

Another test: after shutting down your car after a regular drive, try depressing the brake to the floor a few times. You'll soon exhaust the boost reservoir and the brake pedal will become very stiff now that it's fallen back to full hydraulic operation. In this condition if you hold the pedal halfway down when you start the car you'll feel the brake boost kick in soon as the engine starts.

Note that in many engines, vacuum is provided by a separate vacuum pump, which is driven by the rotation of the engine or rarely by an electric motor. This is always true for a diesel engine (because there is no intake manifold vacuum) and many petrol engines too (if they are direct injection, they also may not have sufficient intake manifold vacuum).

Even if you have a total engine failure, if you're going down a hill, if you keep it in gear then the motion will be turning the engine still, which will be generating vacuum regardless of whether that is by a separate vacuum pump or the intake manifold (unless the throttle valve also fails or you get a gaping hole in the side of your intake, but that seems unlikely).

So braking should still keep working as normal, as long as you keep it in gear and don't disengage the clutch.

TIL! Haven't worked with anything that new.

Note that Toyota only introduced vacuum pumps in 2017, well after the changeover to electronic throttles. All plotted unintended acceleration complaints would've been without a separate vacuum pump.