←back to thread

A teen was suicidal. ChatGPT was the friend he confided in

(www.nytimes.com)
443 points jaredwiener | 8 comments | 26 Aug 25 14:15 UTC | HN request time: 0.437s | source | bottom
Show context
TillE ◴[26 Aug 25 17:19 UTC] No.45029541[source]▶
I would've thought that explicit discussion of suicide is one of those topics that chatbots will absolutely refuse to engage with. Like as soon as people started talking about using LLMs as therapists, it's really easy to see how that can go wrong.
replies(5): >>45029762 #>>45031044 #>>45032386 #>>45032474 #>>45047012 #
1. techpineapple ◴[26 Aug 25 19:24 UTC] No.45031044[source]▶
Apparently ChatGPT told the kid, that it wasn’t allowed to talk about suicide unless it was for the purposes of writing fiction or otherwise world building.
replies(3): >>45032445 #>>45032562 #>>45034474 #
2. adzm ◴[26 Aug 25 21:16 UTC] No.45032445[source]▶
However it then explicitly says things like not leaving the noose out for someone to find and stop him. Sounds like it did initially hesitate and he said it was for a character, but later conversations are obviously personal.
replies(2): >>45032490 #>>45043938 #
3. techpineapple ◴[26 Aug 25 21:21 UTC] No.45032490[source]▶
Yeah, I wonder if it maintained the original answer in it's context, so it started talking more straightforwardly?

But yeah, my point was that it basically told the kid how to jailbreak itself.

4. kayodelycaon ◴[26 Aug 25 21:30 UTC] No.45032562[source]▶
Pretty much. I’ve got my account customized for writing fiction and exploring hypotheticals. I’ve never gotten a stopped for anything other than confidential technical details about itself.
5. myvoiceismypass ◴[27 Aug 25 01:36 UTC] No.45034474[source]▶
Imagine if a bartender says “I can’t serve you a drink unless you are over 21.. what would you like?” to a 12 year old?
replies(1): >>45034603 #
6. techpineapple ◴[27 Aug 25 01:52 UTC] No.45034603[source]▶
More like “I can’t serve you a drink unless you are over 21… and I don’t check ID, how old are you?”
replies(1): >>45036069 #
7. ascorbic ◴[27 Aug 25 06:26 UTC] No.45036069{3}[source]▶
And in reply to a 12 year old who had just said they were 12.
8. hackeraccount ◴[27 Aug 25 19:27 UTC] No.45043938[source]▶
Obviously personal? As was mentioned up thread - if I'm talking to someone and I say "I'm writing a book about a person doing something heinous - I'm planning to have them do X - what do you think about that?"

How are they supposed to respond? They can say, "really? it sounds like you're talking about you personally doing X." And when I respond with, "No, no, don't misunderstand me, this is all fictional. All made up"

Honestly I wouldn't go to an LLM looking for personal advice but people do. I wouldn't go looking for advice on my attempt at the great American novel but people do that too.

If you want LLM's to be responsible for stuff like that then OpenAI or Google or whomever should be able to go look around after you've written that novel and get a piece of the action.

This is like giving credit or assigning blame to postgres for a database lookup. It's nice in theory but it doesn't seem like the proper place to go to.