←back to thread

The Therac-25 Incident (2021)

(thedailywtf.com)
449 points lemper | 1 comments | 27 Aug 25 06:57 UTC | HN request time: 0.218s | source
Show context
isopede ◴[27 Aug 25 08:23 UTC] No.45036862[source]▶
I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone.

Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a radiation machine.

replies(6): >>45036933 #>>45036958 #>>45037102 #>>45037245 #>>45037729 #>>45042356 #
SCdF ◴[27 Aug 25 09:05 UTC] No.45037102[source]▶
The Horizon (UK Royal Mail accounting software) incident killed multiple postmasters through suicide, and bankrupted and destroyed the lives of dozens or hundreds more.

The core takeaway developers should have from Therac-25 is not that this happens just on "really important" software, but that all software is important, and all software can kill, and you need to always care.

replies(2): >>45037211 #>>45037542 #
maweki ◴[27 Aug 25 10:07 UTC] No.45037542[source]▶
But there is still a difference here. Provenance and proper traceability would have allowed the subpostmasters to show their innocence and prove the system failable.

In the Therac-25 case, the killing was quite immediate and it would have happened even if the correct radiation dose was recorded.

replies(2): >>45038610 #>>45040814 #
1. scott_w ◴[27 Aug 25 12:14 UTC] No.45038610[source]▶
I’m not sure it would. Remember that the prosecutors in this case were outright lying to the courts about the system! When you hit that point, it’s really hard to even get a clean audit trail out in the open any more!