←back to thread

Why language models hallucinate

(openai.com)
277 points simianwords | 2 comments | 06 Sep 25 07:41 UTC | HN request time: 0.599s | source
Show context
roxolotl ◴[06 Sep 25 13:09 UTC] No.45148981[source]▶
This seems inherently false to me. Or at least partly false. It’s reasonable to say LLMs hallucinate because they aren’t trained to say they don’t have a statistically significant answer. But there is no knowledge of correct vs incorrect in these systems. It’s all statistics so what OpenAI is describing sounds like a reasonable way to reduce hallucinations but not a way to eliminate them nor the root cause.
replies(4): >>45149040 #>>45149166 #>>45149458 #>>45149946 #
mountainriver ◴[06 Sep 25 15:08 UTC] No.45149946[source]▶
There is knowledge of correct and incorrect, that’s what loss is, there are just often many possible answers to a question.

This is the same reason that RLVR works. There is just right one answer and LLMs learn this fairly well but not perfectly (yet)

replies(1): >>45151002 #
1. Jensson ◴[06 Sep 25 17:08 UTC] No.45151002[source]▶
> There is knowledge of correct and incorrect, that’s what loss is

Loss is only correctness in terms of correct language, not correct knowledge. It correlates with correct knowledge, but that is all, that correlation is why LLM is useful for tasks at all but we still don't have a direct measure for correct knowledge in the models.

So for language tasks loss is correctness, so for things like translations LLM are extremely reliable. But for most other kinds of tasks they are just loosely correlated.

replies(1): >>45154040 #
2. mountainriver ◴[07 Sep 25 00:08 UTC] No.45154040[source]▶
We do with RLVR and that works, there is only one answer, it has to find it. LLMs are often also trained on factual information, and tested on that.

If the knowledge can be represented in text then they can learn it, if it can't then we need a multimodal model.