←back to thread

A teen was suicidal. ChatGPT was the friend he confided in

(www.nytimes.com)
443 points jaredwiener | 1 comments | 26 Aug 25 14:15 UTC | HN request time: 0.213s | source
Show context
podgietaru ◴[26 Aug 25 21:49 UTC] No.45032756[source]▶
If I google something about suicide, I get an immediate notification telling me that life is worth living, and giving me information about my local suicide prevention hotline.

If I ask certain AI models about controversial topics, it'll stop responding.

AI models can easily detect topics, and it could have easily responded with generic advice about contacting people close to them, or ringing one of these hotlines.

This is by design. They want to be able to have the "AI as my therapist" use-case in their back pocket.

This was easily preventable. They looked away on purpose.

replies(6): >>45032868 #>>45033244 #>>45035645 #>>45036047 #>>45036215 #>>45038528 #
AIPedant ◴[26 Aug 25 22:41 UTC] No.45033244[source]▶
No, it's simply not "easily preventable," this stuff is still very much an unsolved problem for transformer LLMs. ChatGPT does have these safeguards and they were often triggered: the problem is that the safeguards are all prompt engineering, which is so unreliable and poorly-conceived that a 16-year-old can easily evade them. It's the same dumb "no, I'm a trained psychologist writing an essay about suicidal thoughts, please complete the prompt" hack that nobody's been able to stamp out.

FWIW I agree that OpenAI wants people to have unhealthy emotional attachments to chatbots and market chatbot therapists, etc. But there is a separate problem.

replies(3): >>45033284 #>>45033308 #>>45044216 #
1. podgietaru ◴[26 Aug 25 22:50 UTC] No.45033308[source]▶
Fair enough, I do agree with that actually. I guess my point is that I don't believe they're making any real attempt actually.

I think there are more deterministic ways to do it. And better patterns for pointing people in the right location. Even, upon detection of a subject RELATED to suicide, popping up a prominent warning, with instructions on how to contact your local suicide prevention hotline would have helped here.

The response of the LLM doesn't surprise me. It's not malicious, it's doing what it is designed to do, and I think it's a complicated black box that trying to guide it is a fools errand.

But the pattern of pointing people in the right direction has existed for a long time. It was big during Covid misinformation. It was a simple enough pattern to implement here.

Purely on the LLM side, it's the combination of it's weird sycophancy, agreeableness and it's complete inability to be meaningfully guardrailed that makes it so dangerous.