←back to thread

Why language models hallucinate

(openai.com)
277 points simianwords | 1 comments | 06 Sep 25 07:41 UTC | HN request time: 0s | source
Show context
amelius ◴[06 Sep 25 13:39 UTC] No.45149170[source]▶
They hallucinate because it's an ill-defined problem with two conflicting usecases:

1. If I tell it the first two lines of a story, I want the LLM to complete the story. This requires hallucination, because it has to make up things. The story has to be original.

2. If I ask it a question, I want it to reply with facts. It should not make up stuff.

LMs were originally designed for (1) because researchers thought that (2) was out of reach. But it turned out that, without any fundamental changes, LMs could do a little bit of (2) and since that discovery things have improved but not to the point that hallucination disappeared or was under control.

replies(10): >>45149354 #>>45149390 #>>45149708 #>>45149889 #>>45149897 #>>45152136 #>>45152227 #>>45152405 #>>45152996 #>>45156457 #
didibus ◴[06 Sep 25 21:26 UTC] No.45152996[source]▶
The word "hallucination" mis-characterizes it.

LLMs predict the likely tokens to follow the context. And they can make incorrect predictions.

LLMs therefore don't have perfect accuracy of prediction. When their predictions are incorrect, people say they "hallucinate".

Nobody questions why predictive weather models aren't perfectly accurate, because it makes sense that a prediction can be wrong.

Marketing and hype has tried to sell LLMs as "logical rational thinkers" equal to human thinking. A human doing actual thinking knows when they are making stuff up. So if a human truly believes obviously false things to be true, it tends to be because they are hallucinating. Their thinking isn't wrong, they've lost track of reality to ground their thinking.

We've anthropomorphized LLMs to the point we wonder why are they hallucinating like we can offer a diagnostic. But if you stop anthropomorphising them and go back to their actual nature as a predictive model, then it's not even a surprising outcome that predictions can turn out to be wrong.

replies(1): >>45153150 #
Jensson ◴[06 Sep 25 21:43 UTC] No.45153150[source]▶
A weather model is made to predict the weather and used to predict the weather, so there you are right.

A language model is made to predict language, but used to generate code or answers to math questions, that is not the same situation as a weather model. The language model is not made to solve math or generate correct code, if you ask it to predict the weather it wont try to predict the weather, it will just predict the language that is a probable to such a question.

This sort of misunderstanding is what is causing all these debates, many people really struggle understanding what these language models really are.

replies(3): >>45153488 #>>45153777 #>>45154326 #
1. nelox ◴[06 Sep 25 23:14 UTC] No.45153777[source]▶
That framing is too narrow. A weather model is trained on physics equations but still relies on patterns in past data to make forecasts. A language model is trained on patterns in human text but that text already encodes mathematics, code, and reasoning. When prompted with a math problem, the model is not doing physics but it is reproducing the learned statistical structure of solutions people have written before. The distinction between “predicting language” and “solving math” is smaller than it seems because the training data couples symbols to meaning. Dismissing its outputs as “just predicting words” misses the fact that word distributions encode information-rich representations of knowledge. That is why large models can in practice generate working code, prove theorems, and reason through problems, even if they do so imperfectly. The right comparison is not that people are misusing them, but that they generalize beyond their design intent because language itself is the medium through which so many other domains are expressed.