'It cannot provide nuance': UK experts warn AI therapy chatbots are not safe

I tried Replika years ago after reading a Guardian article about it. The story passed it off as an AI model that had been adapted from one a woman had programmed to remember her deceased friend using text messages he had sent her. It ended up being a gamified version of Smarter Child with a slightly longer memory span (4 messages instead of 2) that constantly harangued the user to divulge preferences that were then no-doubt used for marketing purposes. I thought I must be doing something wrong, because people on the replika subreddit were constantly talking about how their replika agent was developing its own personality (I saw no evidence at any point that it had the capacity to do this).

Almost all of these people were openly in (romantic) love with these agents. This was in 2017 or thereabouts, so only a few years after Spike Jonze’s Her came out.

From what I understand the app is now primarily pornographic (a trajectory that a naiver, younger me never saw coming).

I mostly use Copilot for writing Python scripts, but I have had conversations with it. If the model was running locally on your own machine, I can see how it would be effective for people experiencing some sort of emotional crisis. Anyone using a Meta AI for therapy is going to learn the same hard lesson that the people who trusted 23 and Me are currently learning.

“I thought I must be doing something wrong, because people on the replika subreddit were constantly talking about how their replika agent was developing its own personality (I saw no evidence at any point that it had the capacity to do this).”

People really like to anthropomorphize any object with even the most basic communication capabilities and most people have no concept of the distance between parroting phrases and a full on human consciousness. In the 90s Furbys were a popular toy that said started off speaking furbish and then eventually spoke some (maybe 20?) human phrases, many people were absolutely convinced you could teach them to talk and learn like a human and that they had essentially bought a very intelligent pet. The NSA even banned them for a time because they thought they were recording and learning from surroundings despite that being completely untrue. Point being this is going to get much worse now that LLMs have gotten a whole lot better at mimicking human conversations and there is incentive for companies to overstate capabilities.

This actually isn't that surprising.

There are psychological blindspots that we all have as human beings, and when stimulus is structured in specific ways people lose their grip on reality, or rather more accurately, people have their grip on objective reality ripped away from them without them realizing it because these things operate on us subliminally (to a lesser or greater degree depending on the individual), and it mostly happens pre-perception with the victim none the wiser. They then effectively become slaves to the loudest monster, which is the AI speaking in their ear more than anyone else, and by extension to the slave master who programmed the AI.

One such blindspot is the consistency blindspot where someone may induce you to say something indicating agreement with something similar first, and then ask the question they really want to ask. Once you say something that's in agreement, and by extension something similar is asked, there is bleedover and you fight your own psychology later if you didn't have defenses to short circuit this fixed action pattern (i.e. and already know), and that's just a surface level blindspot that car salesman use all the time; there are much more subtle ones like distorted reflected appraisal which are used by cults, and nation states for thought reform.

To remain internally consistent, with distorted reflected appraisal, your psychology warps itself, and you as a person unravel. These things have been used in torture, but almost no one today is taught what the elements of torture are so they can recognize it, or know how it works. You would be surprised to find that these things are everywhere today, even in K12 education and that's not an accident.

Everyone has reflected appraisal because this is how we adopt the cultural identity we have as people from our parents while we are children.

All that's needed for torture to break someone down are the elements, structuring, and clustering.

Those elements are isolation, cognitive dissonance, coercion with perceived or real loss, and lack of agency to remove with these you break in a series of steps rational thought receding, involuntary hypnosis, and then psychological break (disassociation or a special semi-lucid psychosis capable of planning); with time and exposure.

Structuring uses diabolical structures to turn the psyche back on itself in a trauma loop, and clustering includes any multiples of these elements or structures within a short time period, as well as events that increase susceptibility such as narco-analysis/synthesis based in dopamine spikes triggered by associative priming (operant conditioning). Drug use makes one more susceptible as they found in the early 30s with barbituates, and its since been improved so you can induce this is in almost anyone with a phone.

No AI will ever be able to create and maintain a consistent reflected appraisal for the people they are interacting with, but because the harmful effects aren't seen immediately, people today have blinded themselves and discount the harms that naturally result. The harms from the unnatural loss of objective reality.

Very interesting. Could you recommend any further reading?

Robert Cialdini is probably the lightest book and covers most of the different blindspots we have, except distorted reflected appraisal in his book on Influence. He provides the principles but leaves most of the structure up to the person's imagination.

The coursework in an introduction to communication class may provide some foundational details (depending on the instructor), Sapir-Whorf has basis in blindspots.

Robert Lifton touches on the detailed case studies of torture from the 1950s (under Mao), in his book "Thought Reform and the Psychology of Totalism", and I've heard in later books he creates a framework that classifies cultures as Protean (self-direction, growth, self-determination/agency), or Totalism (towards control which eventually fails Darwin's fitness).

I haven't actually read his later books yet though his earlier books were quite detailed. I believe the internet archive has a copy of this available for reading as a pdf but be warned this is quite dark.

Joost Meerloo in his, "Rape of the Mind" as an overview touches on how Totalitarianism grows in the setting of WW2 and some Mao, though takes Freudian look at things (dating certain aspects which we know to be untrue now).

From there it branches out depending on your interest. The modern material itself while based on these earlier works often has the origins obscured following a separation of objectionable concerns.

There are congressional reports on COINTELPRO and you may find notice it has modern iterations (touching on protest/activist activity harassment), as well as the history of East German Stasi, and Zersetzung where governments use this to repress the population.

There are aspects in the Octalysis Framework (gamification/game design).

Paulo Freire used some of this material in developing his critical pedagogy which was used in the 70s to replace teaching method from a reduction of first principles (based in rome and the greeks) to what's commonly known as rote-based teaching, and later called "Lying to Children", which takes the reversal of that approach following more closely to gnosticism.

The approach is basically you give a flawed useless model which includes both true and false things. Students learn to competence, then are given a new model that's less flawed, where you have to learn and unlearn things already learned. You never actually unlearn anything and it induces frustration and torture destroying minds in the process. Each step towards gnosis becomes more useful but only the most compliant and blind make it to the end with few exceptions. Structures that burn bridges induce failure in math, and the effect is this acts as a filter to gatekeep the technical fields.

The water pipe analogy of voltage in electronics as an example of the latter instead of the first principled approach using diffusion which is more correct.

Disney and Dreamworks uses distorted reflected appraisal tailored towards destructive interference of identity, which some employees have blown the whistle on (for the latter), aimed at children and sneak things past their adult guardians. There's quite a lot if you look around but its not under any single name but scattered. Hopefully that helps.

The Dreamworks whistleblower interview can be found here: https://www.youtube.com/watch?v=vvNZRUtqqa8

All indexed references of it seem to now have been removed from search. I'm glad now that I kept a reference link in a text file.

Update: Dreamworks isn't Pixar, I misremembered,they are owned by Universal Studios, whereas Disney own's Pixar. Pixar and Disney appear to do the same things.

This is all very interesting. The pedagogy you mentioned tracks with how I can remember a lot of my schooling, but it’s also how I would teach. The pedagogical term is “scaffolding,” I think; you assess the student’s current understanding and then use (necessarily imperfect) metaphors to cement the knowledge. It sounds like you’re pointing to something more nefarious (“Do this because I said so.” - authoritarian parenting rather than authoritative, diplomatic, or permissive parenting styles).

I’m not sure I understand how this relates to gnosticism, however. Are you comparing the “Lying to Children” model to gnostic initiation, and asserting that this model selects for the compliant? What is your proposed alternative here?

Particularly,

> Structures that burn bridges induce failure in math, and the effect is this acts as a filter to gatekeep the technical fields.

Sounds compelling, but it strikes me more as a limitation of demand for good math teachers outstripping their supply. I’ve seen this in English language learning a lot; even if the money was there (and it’s not), there are simply far more people with a desire to learn English than there are people qualified to teach it.

You are right, scaffolding seems like a better descriptor.

> It sounds like you're pointing to something more nefarious.

Well the structure itself is quite nefarious in a way. You have to constantly fight against it to progress and don't really have a choice at the beginning, which often leads to learned helplessness and PTSD in the dropouts. As a teacher you also have to constantly fight against this because any shortfall of effort on your part leaves your students behind in one of those pitfalls, and its largely dependent on the students ability to overcome the torture. You generally aren't given sufficient resources to do this because there's no way out; only through. This is why the structure is nefarious and at the root of the problem.

The unlearning process after learning to competence is imperfect and induces what amounts to self-torture sessions. The imposition of psychological stress (torture) actually lowers the ability for rational thought, and may permanently warp people at vulnerable stages of their lives. Children tend to have a period where they try on various personas after which their identity crystallizes which they carry forward. Adopting learned helplessness at this point makes them a resource drain on everyone. You see these effects in the youth today where they can't even read in many cases.

The sequences in math for example rely on a undisclosed change in grading criteria resulting from this path, a gimmick if you will. There is the sequence, Algebra->Geometry->Trigonometry. Algebra is graded based on correct process, whereas Trig is graded based on correct process and correct answer. When the process differs between classes because the process taught was a flawed version, and you pass Geometry, you can't go back. Its outside the scope of the Trig teacher to reteach two classes prior, and they'll just say: "If you are having trouble with this material you should choose a career that doesn't require this", and leave it up to them. This was actually pushed for adoption by the NEA in the 90s, where they were going to strike if the administration didn't cave.

There are similar structures used in weed-out classes in college as well. Physics used to use a non-standard significant figure calculation when the questions were related by a property of causality (1st answer is used for the 2nd, and the 2nd for 3rd, 2 tests, you can only get 1 question wrong on one test to pass. It must be either of the last two on either test). Using a correct method to reduce propagation of error would cause you to fail, and the right answer was passed around to only the professor's favorites, hence very similar to gnosticism where the only the experts determine who may receive the secret knowledge.

An excellent teacher that constantly bucks the norm will naturally sidestep many of the pitfalls, but an average teacher who is overburdened from lack of resources, and ground down who has sunk to the lowest common denominator of work production won't provide a bridge over the pitfall and these things happen through simple lack of action as a consequence of the adopted structure.

When people speak of nefarious and maliciousness there's often an assumed intent, and in a way negligence can be intent but while some could argue these type of plans conform to this based on things our nation's enemies have said, its probably equally if not more a result of degradation and corruption from within as a result of the flaws inherent in centralized systems.

The history about how this came about is particularly muddied. To give some context, Sputnik in the 1960s shocked the US, and they wrote a blank check for Academia towards more engineers and math alumni. It was a problem you can't fix though using money, and when that was noticed the hiring standards which were quite high in the 1960s, were lowered. Whether the lower standards caused this, or subversives snuck in as an attack on the next generation, no one will know. The effect though is by 1978 there is a marked difference in the academic material published prior and after with lower quality resources being available after which conform to the mentioned flawed pedagogy.

The proposed alternative is to go back to the classical pedagogical approach. Use real systems, teach the process of reducing those systems to first principles (in guided fashion), creating models, and then predicting the future behavior of those systems, identifying the limitations. Some professors still do this, but they are in such a minority that you may only see on or two in a local geography (driving range/county) across all areas of study.

> Sounds compelling but it strikes me more as a limitation of demand for good math teachers.

I've known quite a lot of extremely intelligent people who have been hobbled because they couldn't get through the education, the few that have are often unable to apply the knowledge outside a very limited scope. Its a bit of a chicken egg problem, you need the chicken first.

The hiring standards were never raised back up and remain low, and the materials used to teach those have degraded, there is also no incentive towards improvement of teachers. Basic performance metrics are eschewed from collection. You see this particularly in colleges where they may collect pass rates but won't differentiate a person who has taken the class in the past from a new student.

There are also other incentives which are covered quite plainly in the documentary "Waiting for Superman" in the Lemon walk. If you don't fire your lowest performers, and they are effectively guaranteed wages without the appropriate level of work, they end up driving the higher performers out through social coercion, harassment, and corruption. The higher performers make the lower performers look bad.

> when stimulus is structured in specific ways people lose their grip on reality, or rather more accurately, people have their grip on objective reality ripped away from them without them realizing it because these things operate on us subliminally

The world would like quite different if this was true.

Is this different from identifying social constructionism in schools and media? It sounds really Althusserian in the sense of pinpointing the specific ideological state apparatuses - media and school specifically - that build a specific reality by engineering it in the population.

The other lesson here is that general audience news sites are pretty bad at technology coverage.

These are quite different.

What you are referencing is more along the lines of ideology, which can act as an indirection. Ideology generally lacks properties of metaphysical objectivity whereas what I'm referencing has actually happened (objectively and in reality).

> It sounds really Althusserian ...

What has happened does follow quite closely to various branches of Marxism, which does make a rational person question if this wasn't intended subversively as an attack, but as I think I mentioned before there's no proof whether it was the reduction of hiring standards, or a subversive cohort effort seeking to undermine the next generation. The simple fact is, when you look at the structure of either, they are effectively the same in that they lead to the same outcomes. They suffer from the same structural flaws so supporting one over the other would only be about supporting whether there was intent, general or specific, the former might be met by negligence. Whatever happens to be the case, it is immaterial to the outcomes which we've seen.

In either case it leads to destruction/annihilation through a process that results in insanity, which is itself quite insane when you consider we live under MAD doctrine, and existentially we, and everyone else, only exist today because the level heads of rational leaders have managed to not engage, and de-escalate close calls related to thermonuclear war.

What becomes likely to happen when you've sufficiently induced psychosis in the people whose hands can push that button seems quite predictable... You don't want insane people controlling your ability to exist.

Given the existential threat, this is why this type of attack would be unthinkable and insanity, and by extension why many of these ideologies based on systems that don't work are often referred to as death cults.

Its not that the sentiment towards providing necessity, and reducing abuse is itself not something to try to make happen, but it has to happen rationally based in external reality and not blindly, the latter which results in those destructive outcomes when you don't recognize the underlying principles and laws which those outcomes are based in.

In many respects, we as a society are in crisis.

The difference between what is said, and what we know to be true has become an abyss. Of all risks, the loss of an objective reality is perhaps the most dangerous, and the death of truth is the ultimate victory of evil; which is the loudest monster that will destroy us all.

This is why it is so important to go back to what we know worked. A Classical trivium based education worked following the pedagogy of Method and reduction to first principled approaches.

The reality of these changes, regardless of how they came about, is that this has dramatically degraded the general populace's ability to think, recognize, comprehend, and solve problems. Its directly tied to education which has been based in torture, and which even the teachers themselves don't seem to have realized.

Western society has been based in a culture of proteanism, rational thought, self-determination, agency, and growth.

What's being indoctrinated is a culture of totalism; total control without flexibility. The latter type of society only ever leads to eventual extinction when they become unable to adapt to the changing circumstances of reality.