←back to thread

Why language models hallucinate

(openai.com)
277 points simianwords | 4 comments | 06 Sep 25 07:41 UTC | HN request time: 0.001s | source
Show context
amelius ◴[06 Sep 25 13:39 UTC] No.45149170[source]▶
They hallucinate because it's an ill-defined problem with two conflicting usecases:

1. If I tell it the first two lines of a story, I want the LLM to complete the story. This requires hallucination, because it has to make up things. The story has to be original.

2. If I ask it a question, I want it to reply with facts. It should not make up stuff.

LMs were originally designed for (1) because researchers thought that (2) was out of reach. But it turned out that, without any fundamental changes, LMs could do a little bit of (2) and since that discovery things have improved but not to the point that hallucination disappeared or was under control.

replies(10): >>45149354 #>>45149390 #>>45149708 #>>45149889 #>>45149897 #>>45152136 #>>45152227 #>>45152405 #>>45152996 #>>45156457 #
wavemode ◴[06 Sep 25 14:04 UTC] No.45149354[source]▶
Indeed - as Rebecca Parsons puts it, all an LLM knows how to do is hallucinate. Users just tend to find some of these hallucinations useful, and some not.
replies(5): >>45149571 #>>45149593 #>>45149888 #>>45149966 #>>45152431 #
throwawaymaths ◴[06 Sep 25 14:28 UTC] No.45149571[source]▶
that's wrong. there is probably a categorical difference between making something up due to some sort of inferential induction from the kv cache context under the pressure of producing a token -- any token -- and actually looking something up and producing a token.

so if you ask, "what is the capital of colorado" and it answers "denver" calling it a Hallucination is nihilistic nonsense that paves over actually stopping to try and understand important dynamics happening in the llm matrices

replies(3): >>45149984 #>>45152027 #>>45152539 #
1. saghm ◴[06 Sep 25 19:09 UTC] No.45152027[source]▶
> so if you ask, "what is the capital of colorado" and it answers "denver" calling it a Hallucination is nihilistic nonsense that paves over actually stopping to try and understand important dynamics happening in the llm matrices

On the other hand, calling it anything other than a hallucination misrepresents the idea of truth as being something that these models have any ability to differentiate between their outputs based on whether they accurately reflect reality by conflating a fundamentally unsolved problem as an engineering tradeoff.

replies(1): >>45152907 #
2. ComplexSystems ◴[06 Sep 25 21:15 UTC] No.45152907[source]▶
It isn't a hallucination because that isn't how the term is defined. The term "hallucination" refers, very specifically, to "plausible but false statements generated by language models."

At the end of the day, the goal is to train models that are able to differentiate between true and false statements, at least to a much better degree than they can now, and the linked article seems to have some very interesting suggestions about how to get them to do that.

replies(2): >>45153078 #>>45166613 #
3. throwawaymaths ◴[06 Sep 25 21:36 UTC] No.45153078[source]▶
your point is good and taken but i would amend slightly -- i dont think that "absolute truth" is itself a goal, but rather "how aware is it that it doesn't know something". this negative space is frustratingly hard to capture in the llm architecture (though almost certainly there are signs -- if you had direct access to the logits array, for example)
4. player1234 ◴[08 Sep 25 10:29 UTC] No.45166613[source]▶
Why use a word that you have to redefine the meaning of? The answer is to deceive.