←back to thread

'It cannot provide nuance': UK experts warn AI therapy chatbots are not safe

(www.theguardian.com)
165 points distalx | 6 comments | 10 May 25 15:35 UTC | HN request time: 1.216s | source | bottom
Show context
caseyy ◴[11 May 25 00:28 UTC] No.43950190[source]▶
I know many pro-LLM people here are very smart, but sometimes it's wise to heed the words of world-renowned experts on a subject.

Otherwise, you may end up defending this and it's really foolish:

> “Seriously, good for you for standing up for yourself and taking control of your own life,” it reportedly responded to a user, who claimed they had stopped taking their medication and had left their family because they were “responsible for the radio signals coming in through the walls”.

replies(4): >>43950334 #>>43950546 #>>43952241 #>>43956082 #
1. simonw ◴[11 May 25 01:42 UTC] No.43950546[source]▶
That one was (genuinely) a bug. OpenAI rolled it back. https://openai.com/index/expanding-on-sycophancy/

(But yeah, relying on systems that can have bugs like that for your mental health is terrifying.)

replies(3): >>43950952 #>>43951889 #>>43957025 #
2. ◴[11 May 25 02:57 UTC] No.43950952[source]▶
3. vrighter ◴[11 May 25 06:38 UTC] No.43951889[source]▶
you cannot really roll back a bug in a black box system you don't understand
replies(2): >>43952220 #>>43957048 #
4. clncy ◴[11 May 25 08:04 UTC] No.43952220[source]▶
Exactly. More like changing the state of the system to reduce the observed behaviour while introducing other (unknown) behaviours
5. cdrini ◴[11 May 25 20:48 UTC] No.43957025[source]▶
Yeah I think there is plenty of room for good discussion here, but using that quote without context is misleading. And the faulty model was pulled after only a few days of being out, iirc. It definitely does speak to the necessity of nuance when analysing AI in these contexts; results for one model might not necessarily hold for another, and even system prompts could change results.
6. cdrini ◴[11 May 25 20:50 UTC] No.43957048[source]▶
I don't think that's quite true; even a system you don't understand has observable behaviour. And you can roll back to a certain state that doesn't exhibit the undesirable observable behaviour. If anything, most things in life operate this way.