A teen was suicidal. ChatGPT was the friend he confided in

This is a clear example of why the people claiming that using a chatbot for therapy is better than no therapy are... I'll be extremely generous and say misguided. This kid wanted his parents to know he was thinking about this and the chatbot talked him out of it.

I don't know if it counts as therapy or not but I find the ability to have intelligent (seeming?) conversations with Claude about the most incredibly obscure topics to be very pleasant.

But do you really feel you are conversing? I could never get that feeling. It's not a conversation to me it's just like an on-demand book that might be wrong. Not saying I don't use them to attempt to get information, but it certainly doesn't have a feeling than doing anything other than getting information out of a computer.

"But do you really feel you are conversing?"

Yes. For topics with lots of training data like physics Claude is VERY human sounding. I've had very interesting conversations with Claude Opus about the Boltzmann brain issue and how I feel that the conventional wisdom ignores the low probability of a BBrain having a spatially and temporally consistent set of memories and how the fact that brains existing in a universe that automatically creates consistent memories means the probability of us being Boltzmann brains is very low. Since even if a Boltzmann brain pops into existence its memory will be most likely completely random and completely insane/insensate.

There aren't a lot of people who want to talk about Boltzmann brains.

It sounds like you're mostly just talking to yourself. Which is fine, but being confused about that is where people get into trouble.

"It sounds like you're mostly just talking to yourself"

No, Claude does know a LOT more than I do about most things and does push back on a lot of things. Sometimes I am able to improve my reasoning and other times I realize I was wrong.

Trust me, I am aware of the linear algebra behind the curtain! But even when you mostly understand how they work the best LLMs today are very impressive. And latent spaces fundamentally new way to index data.

You can talk to yourself while reading books and searching the web for information. I don't think the fact that you're learning from information the LLM is pulling in means you're really conversing with it.

I do find LLMs very useful and am extremely impressed by them, I'm not saying you can't learn things this way at all.

But there's nobody else on the line with you. And while they will emit text which contradicts what you say if it's wrong enough, they've been heavily trained to match where you're steering things, even if you're trying to avoid doing any steering.

You can mostly understand how these work and still end up in a feedback loop that you don't realize is a feedback loop. I think this might even be more likely the more the thing has to offer you in terms of learning - the less qualified you are on the subject, the less you can tell when it's subtly yes-and'ing you.