←back to thread

A teen was suicidal. ChatGPT was the friend he confided in

(www.nytimes.com)
443 points jaredwiener | 1 comments | 26 Aug 25 14:15 UTC | HN request time: 0s | source
Show context
TillE ◴[26 Aug 25 17:19 UTC] No.45029541[source]▶
I would've thought that explicit discussion of suicide is one of those topics that chatbots will absolutely refuse to engage with. Like as soon as people started talking about using LLMs as therapists, it's really easy to see how that can go wrong.
replies(5): >>45029762 #>>45031044 #>>45032386 #>>45032474 #>>45047012 #
techpineapple ◴[26 Aug 25 19:24 UTC] No.45031044[source]▶
Apparently ChatGPT told the kid, that it wasn’t allowed to talk about suicide unless it was for the purposes of writing fiction or otherwise world building.
replies(3): >>45032445 #>>45032562 #>>45034474 #
adzm ◴[26 Aug 25 21:16 UTC] No.45032445[source]▶
However it then explicitly says things like not leaving the noose out for someone to find and stop him. Sounds like it did initially hesitate and he said it was for a character, but later conversations are obviously personal.
replies(2): >>45032490 #>>45043938 #
1. hackeraccount ◴[27 Aug 25 19:27 UTC] No.45043938[source]▶
Obviously personal? As was mentioned up thread - if I'm talking to someone and I say "I'm writing a book about a person doing something heinous - I'm planning to have them do X - what do you think about that?"

How are they supposed to respond? They can say, "really? it sounds like you're talking about you personally doing X." And when I respond with, "No, no, don't misunderstand me, this is all fictional. All made up"

Honestly I wouldn't go to an LLM looking for personal advice but people do. I wouldn't go looking for advice on my attempt at the great American novel but people do that too.

If you want LLM's to be responsible for stuff like that then OpenAI or Google or whomever should be able to go look around after you've written that novel and get a piece of the action.

This is like giving credit or assigning blame to postgres for a database lookup. It's nice in theory but it doesn't seem like the proper place to go to.