←back to thread

The Therac-25 Incident (2021)

(thedailywtf.com)
449 points lemper | 3 comments | 27 Aug 25 06:57 UTC | HN request time: 1.109s | source
Show context
benrutter ◴[27 Aug 25 08:18 UTC] No.45036836[source]▶
> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing.

If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think type-systems, unit-testing and defensive-coding can solve all software problems. They definitely can help a lot, but the real failure in the story of the Therac-25 from my understanding, is that it took far too long for incidents to be reported, investigated and fixed.

There was a great Cautionary Tales podcast about the device recently[0], one thing mentioned was that, even aside from the catasrophic accidents, Therac-25 machines were routinely seen by users to show unexplained errors, but these issues never made it to the desk of someone who might fix it.

[0] https://timharford.com/2025/07/cautionary-tales-captain-kirk...

replies(13): >>45036898 #>>45037054 #>>45037090 #>>45037874 #>>45038109 #>>45038360 #>>45038467 #>>45038827 #>>45043421 #>>45044645 #>>45046867 #>>45046969 #>>45047517 #
AdamN ◴[27 Aug 25 08:31 UTC] No.45036898[source]▶
This is true but there also needs to be good developers as well. It can't just be great process and low quality developer practices. There needs to be: 1/ high quality individual processes (development being one of them), 2/ high quality delivery mechanisms, 3/ feedback loops to improve that quality, 4/ out of band mechanisms to inspect and improve the quality.
replies(1): >>45037053 #
Fr3dd1 ◴[27 Aug 25 08:56 UTC] No.45037053[source]▶
I would argue that a good process always has a good self correction mechanism built in. This way, the work done by a "low quality" software developer (this includes almost all of us at some point in time), is always taken into account by the process.
replies(6): >>45037082 #>>45037902 #>>45037927 #>>45038864 #>>45045154 #>>45050022 #
quietbritishjim ◴[27 Aug 25 09:01 UTC] No.45037082[source]▶
Right, but if everyone is low quality then there's no one to do that correction.

That may seem a bit hypothetical but it can easily happen if you have a company that systematically underpays, which I'm sure many of us don't need to think hard to imagine, in which case they will systematically hire poor developers (because those are the only ones that ever applied).

replies(3): >>45037428 #>>45037524 #>>45038431 #
1. ZaoLahma ◴[27 Aug 25 09:51 UTC] No.45037428[source]▶
Replace the "hire poor developers" with "use LLM driven development", and you have the rough outline for a perfect Software Engineering horror movie.

It used to be that the poor performers (dangerous hip-shootin' code commitin' cowpokes) were limited in the amount of code that they could produce per time unit, leaving enough time for others to correct course. Now the cowpokes are producing ridiculous amount of code that you just can't keep up with.

replies(2): >>45045739 #>>45050049 #
2. ponector ◴[27 Aug 25 21:48 UTC] No.45045739[source]▶
Isn't it a future we are moving to? To hire poor (cheap) developers to do LLM-driven development.
3. Cthulhu_ ◴[28 Aug 25 09:09 UTC] No.45050049[source]▶
This is why at every software project I've done in the past 15 odd years, steps were taken to prevent this in an automated and standardized fashion; code reviews of course, but they're more for functionality. Unit test requirements, integration / end-to-end tests based on acceptance criteria, visual regression tests, linting, type systems, OTAP, CI/CD, audit log via Git and standardized commit messages, etc etc etc.

My job hasn't significantly changed with AI, as AI generated code still has to pass all the hurdles I've set up while setting up this project.