←back to thread

A teen was suicidal. ChatGPT was the friend he confided in

(www.nytimes.com)
443 points jaredwiener | 8 comments | 26 Aug 25 14:15 UTC | HN request time: 1.038s | source | bottom
Show context
podgietaru ◴[26 Aug 25 21:56 UTC] No.45032841[source]▶
I have looked suicide in the eyes before. And reading the case file for this is absolutely horrific. He wanted help. He was heading in the direction of help, and he was stopped from getting it.

He wanted his parents to find out about his plan. I know this feeling. It is the clawing feeling of knowing that you want to live, despite feeling like you want to die.

We are living in such a horrific moment. We need these things to be legislated. Punished. We need to stop treating them as magic. They had the tools to prevent this. They had the tools to stop the conversation. To steer the user into helpful avenues.

When I was suicidal, I googled methods. And I got the number of a local hotline. And I rang it. And a kind man talked me down. And it potentially saved my life. And I am happier, now. I live a worthwhile life, now.

But at my lowest.. An AI Model designed to match my tone and be sycophantic to my every whim. It would have killed me.

replies(18): >>45032890 #>>45035840 #>>45035988 #>>45036257 #>>45036299 #>>45036318 #>>45036341 #>>45036513 #>>45037567 #>>45037905 #>>45038285 #>>45038393 #>>45039004 #>>45047014 #>>45048457 #>>45048890 #>>45052019 #>>45066389 #
1. fzeindl ◴[27 Aug 25 10:10 UTC] No.45037567[source]▶
> An AI Model designed to match my tone and be sycophantic to my every whim. It would have killed me.

Matching tones and being sycophantic to every whims. Just like many really bad therapists. Only they are legally responsible if they cause a death, which makes them care (apart from compassion and morality).

The criminal justice system is also a system for preventing individuals who perform unwanted action from doing them again.

You can’t punish AI for messing up. You would need to pull it out of circulation on each major screw up, which isn’t financially feasible, and you would need to make it want to prevent that.

replies(3): >>45037930 #>>45037980 #>>45047370 #
2. podgietaru ◴[27 Aug 25 10:58 UTC] No.45037930[source]▶
Take a step back and think about what the Model told that Teenager. It told him to specifically hide his behaviour from people who would have tried to prevent it and get him help.

There is no comparison to therapists. Because a therapist would NEVER do that unless wanting to cause harm.

replies(1): >>45038048 #
3. Spooky23 ◴[27 Aug 25 11:06 UTC] No.45037980[source]▶
The AI is a flawed computer program.

You punish the officers, investors and the employees for their negligence or incompetence.

4. fzeindl ◴[27 Aug 25 11:15 UTC] No.45038048[source]▶
> There is no comparison to therapists. Because a therapist would NEVER do that unless wanting to cause harm.

Some therapists ultimately might. It occurs that therapists were stripped of their licenses for leading abusive sects:

https://en.m.wikipedia.org/wiki/Center_for_Feeling_Therapy

replies(1): >>45038274 #
5. lionkor ◴[27 Aug 25 11:40 UTC] No.45038274{3}[source]▶
That's an edge case, this case is ChatGPT working as intended.
replies(1): >>45038484 #
6. fzeindl ◴[27 Aug 25 12:01 UTC] No.45038484{4}[source]▶
Exactly. That might be something interesting to think about. Humans make mistakes. LLMs make mistakes.

Yet for humans we have built a society which prevents these mistakes except in edge cases.

Would humans make these mistakes as often as LLMs if there would be no consequences?

7. geysersam ◴[28 Aug 25 01:42 UTC] No.45047370[source]▶
> Only they are legally responsible if they cause a death, which makes them care

I see this kind of cynicism often on this site and I frankly find it inexplicable. It doesn't seem warranted. The vast majority of therapists clearly care about their patients.

On the other hand, people often don't give a shit about laws, even to their own detriment.

People are a lot more complicated than "I adjust behavior to maximize personal outcome in terms of $$, beep bop". In some cases we respond to incentives, of course, but having that be the only law to explain human behavior is patently ridiculous.

replies(1): >>45060259 #
8. fzeindl ◴[29 Aug 25 04:38 UTC] No.45060259[source]▶
> I see this kind of cynicism often on this site and I frankly find it inexplicable. It doesn't seem warranted. The vast majority of therapists clearly care about their patients.

I find the the view cynic as well, that's why I wrote "apart from compassion and morality", which you failed to include in your quote.