←back to thread

Reasoning models reason well, until they don't

(arxiv.org)
215 points optimalsolver | 1 comments | 31 Oct 25 09:23 UTC | HN request time: 0.21s | source
Show context
My_Name ◴[31 Oct 25 11:10 UTC] No.45770715[source]▶
I find that they know what they know fairly well, but if you move beyond that, into what can be reasoned from what they know, they have a profound lack of ability to do that. They are good at repeating their training data, not thinking about it.

The problem, I find, is that they then don't stop, or say they don't know (unless explicitly prompted to do so) they just make stuff up and express it with just as much confidence.

replies(9): >>45770777 #>>45770879 #>>45771048 #>>45771093 #>>45771274 #>>45771331 #>>45771503 #>>45771840 #>>45778422 #
1. Workaccount2 ◴[31 Oct 25 13:36 UTC] No.45771840[source]▶
To be fair, we don't actually know what is and isn't in their training data. So instead we just assign successes to "in the training set" and failures to "not in the training set".

But this is unlikely, because they still can fall over pretty badly on things that are definitely in the training set, and still can have success with things that definitely are not in the training set.