←back to thread

The Therac-25 Incident (2021)

(thedailywtf.com)
449 points lemper | 2 comments | 27 Aug 25 06:57 UTC | HN request time: 0s | source
Show context
isopede ◴[27 Aug 25 08:23 UTC] No.45036862[source]▶
I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone.

Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a radiation machine.

replies(6): >>45036933 #>>45036958 #>>45037102 #>>45037245 #>>45037729 #>>45042356 #
1. the-grump ◴[27 Aug 25 08:37 UTC] No.45036933[source]▶
The 737 MAX MCAS debacle was one such failure, albeit involving a wider system failure and not purely software.

Agreed on the future but I think we were headed there regardless.

replies(1): >>45037086 #
2. jonplackett ◴[27 Aug 25 09:01 UTC] No.45037086[source]▶
Yeah reading this reminded me a lot of MCAS. Though MCAS was intentionally implemented and intentionally kept secret.