←back to thread

Why language models hallucinate

(openai.com)
277 points simianwords | 1 comments | 06 Sep 25 07:41 UTC | HN request time: 0.001s | source
Show context
kingstnap ◴[06 Sep 25 15:51 UTC] No.45150336[source]▶
There is this deeply wrong part of this paper that no one has mentioned:

The model head doesn't hallucinate. The sampler does.

If you ask an LLM when x was born and it doesn't know.

And you take a look at the actual model outputs which is a probability distribution over tokens.

IDK is cleanly represented as a uniform probability Jan 1 to Dec 31

If you ask it to answer a multiple choice question and it doesn't know. It will say this:

25% A, 25% B, 25% C, 25%D.

Which is exactly, and correctly, the "right answer". The model has admitted it doesn't know. It doesn't hallucinate anything.

In reality we need something smarter than a random sampler to actually extract this information out. The knowledge and lack of knowledge is there, you just produced bullshit out of it.

replies(4): >>45150445 #>>45150464 #>>45154049 #>>45167516 #
1. a2128 ◴[08 Sep 25 12:36 UTC] No.45167516[source]▶
This is only true if you have a pretrained base model trained on infinite true data with no bias. In practice it will have picked up some bias, maybe it encountered more famous "James" birthdays in January and on a digit starting with 2, so Jan 2 and Jan 20-29 has a higher probability than all. But finetuning and especially RL completely breaks these probabilities as a measure of certainty because the goals shift from generally modelling text to something else entirely.