←back to thread

The Therac-25 Incident (2021)

(thedailywtf.com)
449 points lemper | 1 comments | 27 Aug 25 06:57 UTC | HN request time: 0s | source
Show context
isopede ◴[27 Aug 25 08:23 UTC] No.45036862[source]▶
I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone.

Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a radiation machine.

replies(6): >>45036933 #>>45036958 #>>45037102 #>>45037245 #>>45037729 #>>45042356 #
1. Maxion ◴[27 Aug 25 08:41 UTC] No.45036958[source]▶
> Personally, I've found even the latest batch of agents fairly poor at embedded systems

I mean even simple crud web apps where the data models are more complex, and where the same data has multiple structures, the LLMs get confused after the second data transformation (at the most).

E.g. You take in data with field created_at, store it as created_on, and send it out to another system as last_modified.