Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a radiation machine.
The core takeaway developers should have from Therac-25 is not that this happens just on "really important" software, but that all software is important, and all software can kill, and you need to always care.
In the Therac-25 case, the killing was quite immediate and it would have happened even if the correct radiation dose was recorded.
> Provenance and proper traceability would have allowed
But there wasn't those things, so they couldn't, so they were driven to suicide.
Bad software killed people. It being slow or fast doesn't seem to matter.
Fast killing software is too fast for that.
It sounds like you're saying that you shouldn't care as much about the quality of "slow killing software" because in theory it can be made better in the future?
But... it wasn't though? Horizon is a real software system that real developers like you and me built that really killed people. The absolutely terrible quality of it was known about. It was downplayed and covered up, including by the developers who were involved, not just the suits.
I don't understand how a possible solution absolves the reality of what was built.
In slowly killing software the audit trail might be faster than the killing. In fast killing software, the audit trail isn't.
My core point is that if you're designing a system, *any system*, you should be thinking about what is required to produce safe software. It isn't just "well I don't work on medical devices that shoot radiation at people, so I don't need to worry"[1]. You still need to worry, you just solve those problems in different ways. It's not just deaths either, it's PII leakage, it's stalking and harassment enablement, it's privilege escalation, etc.
[1] I have heard this, or a variation of this, from dozens of people over the my career. This is my core bug bear about Therac-25, is that it allows people to think this way, and divest themselves of responsibility. I am very happy to hear you are teaching a course about Horizon, because it's a much more grounded example that devs will hopefully see themselves in more. If your course is publicly available btw, I'd love to read it.
And the horizon post office scandal is the last and most poignant example that real people are affected by the systems we build and the design decisions we make. That sometimes easy to forget.