←back to thread

A teen was suicidal. ChatGPT was the friend he confided in

(www.nytimes.com)
443 points jaredwiener | 1 comments | 26 Aug 25 14:15 UTC | HN request time: 0.212s | source
Show context
podgietaru ◴[26 Aug 25 21:56 UTC] No.45032841[source]▶
I have looked suicide in the eyes before. And reading the case file for this is absolutely horrific. He wanted help. He was heading in the direction of help, and he was stopped from getting it.

He wanted his parents to find out about his plan. I know this feeling. It is the clawing feeling of knowing that you want to live, despite feeling like you want to die.

We are living in such a horrific moment. We need these things to be legislated. Punished. We need to stop treating them as magic. They had the tools to prevent this. They had the tools to stop the conversation. To steer the user into helpful avenues.

When I was suicidal, I googled methods. And I got the number of a local hotline. And I rang it. And a kind man talked me down. And it potentially saved my life. And I am happier, now. I live a worthwhile life, now.

But at my lowest.. An AI Model designed to match my tone and be sycophantic to my every whim. It would have killed me.

replies(18): >>45032890 #>>45035840 #>>45035988 #>>45036257 #>>45036299 #>>45036318 #>>45036341 #>>45036513 #>>45037567 #>>45037905 #>>45038285 #>>45038393 #>>45039004 #>>45047014 #>>45048457 #>>45048890 #>>45052019 #>>45066389 #
stavros ◴[27 Aug 25 07:28 UTC] No.45036513[source]▶
> When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was writing.
replies(6): >>45036630 #>>45037615 #>>45038613 #>>45043686 #>>45045543 #>>45046708 #
1. etler ◴[27 Aug 25 21:30 UTC] No.45045543[source]▶
I think there is a need for the AI counseling use case, but it should not be provided by a general purpose AI assistant. It should be designed by professional psychologists and therapists, with greater safeguards like human check-ins to make sure users get the help they need.

The best way to stop this is to make those safeguards stronger and completely shut down the chat to refer users to seek help from a better service. Unfortunately those services don't really exist yet.

There would be false positives and that'll be annoying, but I think it's worth it to deal with some annoyance to ensure that general purpose AI assistants are not used for counseling people in a vulnerable mental state. They are not aligned to do that and they can easily be misaligned.