Most active commenters

    ←back to thread

    The Therac-25 Incident (2021)

    (thedailywtf.com)
    449 points lemper | 11 comments | 27 Aug 25 06:57 UTC | HN request time: 0.001s | source | bottom
    Show context
    isopede ◴[27 Aug 25 08:23 UTC] No.45036862[source]▶
    I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone.

    Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a radiation machine.

    replies(6): >>45036933 #>>45036958 #>>45037102 #>>45037245 #>>45037729 #>>45042356 #
    SCdF ◴[27 Aug 25 09:05 UTC] No.45037102[source]▶
    The Horizon (UK Royal Mail accounting software) incident killed multiple postmasters through suicide, and bankrupted and destroyed the lives of dozens or hundreds more.

    The core takeaway developers should have from Therac-25 is not that this happens just on "really important" software, but that all software is important, and all software can kill, and you need to always care.

    replies(2): >>45037211 #>>45037542 #
    1. hahn-kev ◴[27 Aug 25 09:21 UTC] No.45037211[source]▶
    From what I've read about that incident I don't know what the devs could have done. The company sure was a problem but also the laws basically saying a computer can't be wrong. No dev can solve that problem.
    replies(6): >>45037255 #>>45037256 #>>45037983 #>>45039517 #>>45040795 #>>45044314 #
    2. sim7c00 ◴[27 Aug 25 09:29 UTC] No.45037255[source]▶
    as you point out this was a messup on a lot of levels. its an interesting effect tho not to be dismissed. how your software works and how its perceived and trusted can impact people psychologically.
    3. fuckaj ◴[27 Aug 25 09:29 UTC] No.45037256[source]▶
    Given whole truth testimony?
    replies(1): >>45038580 #
    4. V__ ◴[27 Aug 25 11:06 UTC] No.45037983[source]▶
    > Engineers are legally obligated to report unsafe conduct, activities or behaviours of others that could pose a risk to the public or the environment. [1]

    If software "engineers" want to be taken seriously, then they should also have the obligation to report unsafe/broken software and refuse to ship unsafe/broken software. The developers are just as much to blame as the post office:

    > Fujitsu was aware that Horizon contained software bugs as early as 1999 [2]

    [1] https://engineerscanada.ca/news-and-events/news/the-duty-to-...

    [2] https://en.wikipedia.org/wiki/British_Post_Office_scandal

    replies(2): >>45044340 #>>45050809 #
    5. ◴[27 Aug 25 12:11 UTC] No.45038580[source]▶
    6. siva7 ◴[27 Aug 25 13:38 UTC] No.45039517[source]▶
    Then you haven't read deep enough into the Horizon UK case. The lead devs have to take a major blame for what happened as they lied to the investigators and could have helped prevent early on some suicides if they had courage. These devs are the worst kind of, namely Gareth Jenkins and Anne Chambers.
    7. SCdF ◴[27 Aug 25 15:09 UTC] No.45040795[source]▶
    The code being absolute dog shit was true regardless of that law's existence. There are plenty of things the developers could have done.

    That law is irrelevant to this situation, except in that the lawyers for Fujitsu / Royal Mail used it to imply their code was infallable.

    8. codeulike ◴[27 Aug 25 19:53 UTC] No.45044314[source]▶
    It was a distributed system lashed together by 'consultants' (read: recent graduates with little real world software engineering experience) in an era where best practices around distributed systems were non-existent. They weren't even thinking about what kind of data inconsistencies they might end up with.
    9. simulator5g ◴[27 Aug 25 19:55 UTC] No.45044340[source]▶
    I don't think it's fair to blame individual developers for a systemic failure. Its not their fault there is no governing body to award or remove the title of "software engineer" and promote the concept of a software engineer refusing to do something without harming their career. Other engineering disciplines have laws, lobbied for by their governing body, that protect the ability of individual engineers to prevent higher-ups from making grave mistakes.
    replies(1): >>45046864 #
    10. lmm ◴[28 Aug 25 00:15 UTC] No.45046864{3}[source]▶
    > Its not their fault there is no governing body to award or remove the title of "software engineer" and promote the concept of a software engineer refusing to do something without harming their career.

    Those governing bodies didn't form by magic. If you look at how hostile people on this site are to the idea of unionization or any kind of collective organisation, I'd say a large part of the problem with software is individual developers' attitudes.

    11. donatj ◴[28 Aug 25 11:20 UTC] No.45050809[source]▶
    I have worked in this industry for 20 years and never met a piece of software I would deem "safe". It's all duct tape and spit. All of it.

    I have had software professionally audited by third parties more than a few times, and they basically only ever catch surface level bugs. Recently, the same we the audit finished we independently found a pretty obvious sql injection flaw.

    I think the danger is not in producing unsafe software. The real danger is in thinking it can ever can be safe. It cannot be, and anyone who tells you otherwise is a snake oil salesman.

    If your life depends on software, you are one bit flip from death.