←back to thread

Why language models hallucinate

(openai.com)
277 points simianwords | 1 comments | 06 Sep 25 07:41 UTC | HN request time: 0s | source
Show context
thomasboyer ◴[06 Sep 25 13:50 UTC] No.45149259[source]▶
Great post. Teaching the models to doubt, to say "I don't know"/"I'm unsure"/"I'm sure" is a nice way to make them much better.
replies(2): >>45149845 #>>45150304 #
more_corn ◴[06 Sep 25 14:59 UTC] No.45149845[source]▶
It baffles me that this hasn’t been done yet. Saying I don’t know or I’m unsure is critical for anything that matters.
replies(1): >>45150050 #
1. ACCount37 ◴[06 Sep 25 15:19 UTC] No.45150050[source]▶
Major industry players were doing that for a while now. It's just hard to actually design training regimes that give LLMs better hallucination-avoidance capabilities.

And it's easy to damage the hallucination-avoidance capabilities by training an LLM wrong. As OpenAI has demonstrated when they fried the o3 with RLVR that encouraged guesswork.

That "SAT test incentivizes guesswork" example they give in the article is one they had to learn for themselves the hard way.