'It cannot provide nuance': UK experts warn AI therapy chatbots are not safe

Does it offer 80% of the benefit? An AI could match what a human therapist would say 80% (or 99%) of the time and still provide negative benefit.

Therapy seems like the last place an LLM would be beneficial because it’s very hard to keep an LLM from telling you what you want to hear. I can see anyway you could guarantee that a chatbot cause severe damage to a vulnerable patient by supporting their neurosis.

We’re not anywhere close to an LLM which is trained to be supportive and understanding in tone but will never affirm your irrational fears, insecurities, and delusions.

Sometimes, the process of gathering our thoughts enough to article them into a prompt is where the benefit is. AI as the rubber duck has a lot of value. Understanding that this is what’s needed vs. something deeper, is beyond the scope of what AI can handle.

And that’s fine as long as the person using it has a sophisticated understanding of the technology and a company isn’t selling it as a “therapist”.

When an AI therapist from a health startup confirms that a mentally disturbed person is indeed hearing voices from God, or an insecure teenager uses meta AI as a therapist because Mark Zuckerberg said they should and it agrees with them that yes they are unloveable, then we have a problem.

That last 20% of “missing nuance” is really important if someone is in that state! For the rest of us, the value of an AI therapist roughly matches journaling.

I mean most forms of professional therapy the therapist shouldn't say much at all and certainly shouldn't give advice. The point is to have someone listen in a way that feels like they are really listening

If it's about gathering our thoughts, there's meditation. Or journaling. Or prayer. Some have even claimed that there is an all-powerful being listening to you on the other side with that last one. (One might even call it an intelligence, just not an artificial one.)

There's also talking to a friend. Sure, they could also steer you wrong, but at least they won't be impersonating a therapist, and they won't be doing it to try to please their investors.

Therapists don’t give advice in that they won’t tell you whether you should quit your job, or should you propose to your girlfriend. They will definitely give you basic guidance and confirm that your fears are overblown.

They will not under any circumstances tell you that “yes you are correct, Billy would be more likely to love you if you drop 30 more pounds by throwing up after eating”, but an LLM will if it goes off script.

> most forms of professional therapy the therapist shouldn't say much at all

This is very untrue. Here is a list of psychotherapy modalities: https://en.wikipedia.org/wiki/List_of_psychotherapies. In most (almost every) modalities, the therapist provides an intervention and offers advice (by definition: guidance, recommendations).

There is Carl Rogers' client-centered therapy, non-directive supportive therapy, and that's it for low-intervention modalities off the top of my head. Two out of over a hundred. Hardly "most" at all.

This is very cool. Reading through the list, I discovered:

https://en.m.wikipedia.org/wiki/Person-centered_therapy

That sounds an awful lot like what current gen AIs are capable of.

I believe we are in the very early stages of AI-assisted therapy, much like the early days of psychology itself. Before we understood what was generally acceptable and what was not, it was a Wild West with medical practitioners employing harmful techniques such as lobotomy.

Because there are no standards on what constitutes an emotional support AI, or any agreed upon expectations from them, we can only go by what it seems to be capable of. And it seems to be capable of talking intelligently and logically with deep empathy. A rubber ducky 2.0 that can organize your thoughts and even infer meaning from them on demand.

You can create an LLM to keep a check on the LLM interacting with people. This is basically what all “safety” etc models do - they work as gatekeepers for the more powerful model.

This is an implementation problem and not really a technical limitation. If anything, by focusing on a particular domain (like therapy), the do’s and don’ts become more clear.

Sure you might be able to do that. Or it could turn out that the amount of harmful responses are so varied that trying to block all of them makes the therapy AI useless.

There is a very fine line between being understanding and supportive and enabling bad behavior. I’m not confident that a team of LLMs is going to be able to walk that line consistently anytime soon.

We can’t even get code generating LLMs to stop hallucinating APIs and code is a much narrower domain than therapy.

Telling that you need to make up some BS about LLMs while you say nothing about the many clients who have been assaulted, raped, or killed by their therapist.

How can you so confidently claim that "Therapists will do this and that, they won't do any evil". Did you even read what you posted?

If you could prove that your LLM was only as likely to provide harmful responses as a therapist was to murder you, you might have a point.

For what it’s worth, in my personal experience, ChatGPT 4o, DeepSeek R1, and Grok 3, to an extent, are “smarter” about human behaviour than they are at producing code. There’s likely a lot going on behind the scenes to maintain continuity (so it produces content that’s pretty consistent, at least for me on behavioural discussions), especially with ChatGPT.

It’s been incredibly helpful for my personal use: brainstorming ideas, such as exploring how different scenarios might unfold. For instance, I can ask, “What are the pros and cons of choosing x over y, considering these factors?” or even, “I’m in a tough spot. X and I often argue about Z (provide some background context), and I’m struggling to express my perspective. I’m afraid…” You get the idea.

GPT-4o is remarkably good at putting things in an independent and unbiased third person perspective. It’s definitely not an echo chamber for me. More often, the insights are what I might have come up with if I were observing my own life from a distance.

Now some people have said “sure it’s like journaling”. I think it’s even better, like talking to your journal (ala Harry Potter, like Tom Riddle’s diary) with some level of fact checking (I’ve gotten called out) and human behavioural understanding available at your disposable.

I’m sure they can be very useful for things like that. Provided the user has a sophisticated understanding of the technology as you clearly do. That’s not the same as selling chat bots to vulnerable naive people who think they are talking to an intelligent “therapist”.

And it’s not just code where they go off the rails. If you talk to them for a while they will very frequently end up agreeing with you if you want them to.

I’ve seen this many times when using an LLM to try to learn something new or refresh my memory.