←back to thread

449 points lemper | 2 comments | | HN request time: 0s | source
Show context
isopede ◴[] No.45036862[source]
I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone.

Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a radiation machine.

replies(6): >>45036933 #>>45036958 #>>45037102 #>>45037245 #>>45037729 #>>45042356 #
1. sim7c00 ◴[] No.45037245[source]
talk to anyone in the industries about 'automation' on medical or critical infra devices and they will tell you NO. No touching our devices with your rubbish.

i am pretty confident they wont let claude touch if it they dont even let deterministic automations run...

that being said, maybe there are places. but this is always the sentiment i got. no automating, no scanning, no patching. device is delivered certified and any modifications will invalidate that. any changes need to be validated and certified.

its a different world that makin apps thats for sure.

not to say mistakes arent made and change doesnt happen, but i dont think people designing medical devices will be going yolo mode on their dev cycle anytime soon... give the folks in safety critical system engineering some credit..

replies(1): >>45037662 #
2. throwaway0261 ◴[] No.45037662[source]
> but i dont think people designing medical devices will be going yolo mode on their dev cycle anytime soon

I don't have the same faith in corporate leadership as you, at least not when they see potentially huge savings by firing some of the expensive developers and using AI to write more of the code.