←back to thread

A teen was suicidal. ChatGPT was the friend he confided in

(www.nytimes.com)
443 points jaredwiener | 3 comments | 26 Aug 25 14:15 UTC | HN request time: 0s | source
Show context
rideontime ◴[26 Aug 25 21:03 UTC] No.45032301[source]▶
The full complaint is horrifying. This is not equivalent to a search engine providing access to information about suicide methods. It encouraged him to share these feelings only with ChatGPT, talked him out of actions which would have revealed his intentions to his parents. Praised him for hiding his drinking, thanked him for confiding in it. It groomed him into committing suicide. https://drive.google.com/file/d/1QYyZnGjRgXZY6kR5FA3My1xB3a9...
replies(6): >>45032582 #>>45032731 #>>45035713 #>>45036712 #>>45037683 #>>45039261 #
idle_zealot ◴[26 Aug 25 21:33 UTC] No.45032582[source]▶
I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine learning to avoid accountability.
replies(10): >>45032677 #>>45032798 #>>45032857 #>>45033177 #>>45033202 #>>45035815 #>>45036475 #>>45036923 #>>45037123 #>>45039144 #
edanm ◴[27 Aug 25 08:34 UTC] No.45036923[source]▶
If ChatGPT has helped people be saved who might otherwise have died (e.g. by offering good medical advice that saved them), are all those lives saved also something you "attribute" to OpenAI?

I don't know if ChatGPT has saved lives (thought I've read stories that claim that, yes, this happened). But assuming it has, are you OK saying that OpenAI has saved dozens/hundreds of lives? Given how scaling works, would you be OK saying that OpenAI has saved more lives than most doctors/hospitals, which is what I assume will happen in a few years?

Maybe your answer is yes to all the above! I bring this up because lots of people only want to attribute the downsides to ChatGPT but not the upsides.

replies(3): >>45037014 #>>45037756 #>>45043761 #
nkrisc ◴[27 Aug 25 10:34 UTC] No.45037756[source]▶
In any case, if you kill one person and separately save ten people, you’ll still be prosecuted for killing that one person.
replies(2): >>45038050 #>>45049789 #
mothballed ◴[27 Aug 25 11:16 UTC] No.45038050[source]▶
That's not the standard we hold medical care providers, pharmaceutical companies, or even cops to. Not that I'm saying it would justify it one way or another if we did.
replies(2): >>45038674 #>>45044035 #
Orygin ◴[27 Aug 25 12:21 UTC] No.45038674[source]▶
It absolutely is? If a doctor is responsible for negligence resulting in the death of someone, they don't get a pass because they saved 10 other people in their career.
replies(2): >>45038744 #>>45039427 #
1. ACCount37 ◴[27 Aug 25 13:31 UTC] No.45039427{6}[source]▶
Triage exists. And when the shit hits the fan, it takes the form of "this one isn't worth trying to save, but this one might be".

It's callous and cold, and it results in more lives saved than trying to save everyone.

Does ChatGPT, on the net, save more people than it dooms? Who knows. Plenty of anecdotes both ways, but we wouldn't have the reliable statistics for a long time.

replies(2): >>45039670 #>>45039719 #
2. Orygin ◴[27 Aug 25 13:51 UTC] No.45039670[source]▶
Sure, triage done by a human exist and they are generally absolved from taking that decision that could kill someone (to save others). Because the situation calls for it and would otherwise result in more deaths. But other than the situation calling for it, healthcare professionals are held to a certain standard and can be litigated against for medical malpractice.

That is not the case for ChatGPT apparently, and OpenAI should be held responsible for what their models do. They are very much aware of this because they did fine-tune GPT5 to avoid giving medical advice, even though it's still possible to work around.

3. habinero ◴[27 Aug 25 13:54 UTC] No.45039719[source]▶
Triage exists because in a mass casualty event, medical resources need to go to the most urgent cases first.

Triage is not a punch card that says if you drag 9 people out of a burning building, you can shoot someone on the street for free.