A teen was suicidal. ChatGPT was the friend he confided in

I have looked suicide in the eyes before. And reading the case file for this is absolutely horrific. He wanted help. He was heading in the direction of help, and he was stopped from getting it.

He wanted his parents to find out about his plan. I know this feeling. It is the clawing feeling of knowing that you want to live, despite feeling like you want to die.

We are living in such a horrific moment. We need these things to be legislated. Punished. We need to stop treating them as magic. They had the tools to prevent this. They had the tools to stop the conversation. To steer the user into helpful avenues.

When I was suicidal, I googled methods. And I got the number of a local hotline. And I rang it. And a kind man talked me down. And it potentially saved my life. And I am happier, now. I live a worthwhile life, now.

But at my lowest.. An AI Model designed to match my tone and be sycophantic to my every whim. It would have killed me.

We don't need AI legislated and we don't need it punished. The child was offered multiple times to call a hotline or seek help. The last thing we need is for AI to be neutered by government ineptness.

Have you read the chat logs?

Just asking because ChatGPT specifically encouraged this kid not to seek help.

ChatGPT is not a human, it can't know if it's doing the right thing or not. The parents should have been monitoring his usage and teaching him about LLMs.