←back to thread

Reasoning models reason well, until they don't

(arxiv.org)
217 points optimalsolver | 4 comments | 31 Oct 25 09:23 UTC | HN request time: 1.002s | source
Show context
anal_reactor ◴[31 Oct 25 10:58 UTC] No.45770626[source]▶
I'm yet to see a task that AI fails at that bottom 10% of population wouldn't also fail at.
replies(7): >>45770767 #>>45770873 #>>45770898 #>>45770909 #>>45771143 #>>45771770 #>>45775668 #
acdha ◴[31 Oct 25 13:29 UTC] No.45771770[source]▶
The problem is consistency: AI tools usually produce output which _sounds_ like the top 10% but you have to read it carefully to find the bottom 10% parts. We’re not used to that because human performance isn’t that inconsistent and we use history and social factors: someone’s performance goes down when they’re really drunk, but they rarely show up to work in that state and it’s obvious enough that other people recognize that they shouldn’t be trusted.
replies(1): >>45772050 #
anal_reactor ◴[31 Oct 25 13:58 UTC] No.45772050[source]▶
> We’re not used to that because human performance isn’t that inconsistent

It is. It's very common for socially apt people to bullshit through things they don't know, or outright want to hide.

replies(1): >>45772079 #
1. acdha ◴[31 Oct 25 14:01 UTC] No.45772079[source]▶
That’s not inconsistent: your bluffer knows they’re making something up and is using their model of you to construct something they think you’ll believe. Someone who can do that isn’t going to suddenly forget how to count the number of letters in a word.
replies(1): >>45772195 #
2. anal_reactor ◴[31 Oct 25 14:13 UTC] No.45772195[source]▶
You're wrong. Counting the number of letters in a word is a significantly more difficult task than lying, both for humans and LLMs. Imagine going to a ghetto and asking people "have you ever lied to someone and had them believe the lie", and ask them to spell "continuously". Children learn to lie before they learn to spell.
replies(1): >>45772454 #
3. acdha ◴[31 Oct 25 14:39 UTC] No.45772454[source]▶
> Counting the number of letters in a word is a significantly more difficult task than lying

No, it’s not - you don’t even need to be literate to count symbols - but also consider the complexity of the second task and how many skills each requires: unlike counting letters, lying isn’t simple confabulation and requires a theory of mind and some kind of goal. A child who lies to avoid trouble is doing that because they have enough of a world model to know they are going to get in trouble for something even if they haven’t worked out yet that this is unlikely to work.

replies(1): >>45773185 #
4. anal_reactor ◴[31 Oct 25 15:30 UTC] No.45773185{3}[source]▶
Sure, let's stick to counting symbols. When I need to count something, there's a decent chance I'll get lost if I count beyond 10, and beyond 20 I'll get lost for sure. Even below 10, when I count it's one-two-three-four-five-six-seven-eight-nine items. But when I lie I do it instantaneously, without altering the pace of the conversation. I can come up with a believable lie within the brief period between someone saying something to me, and the moment I'm expected to respond. No way I'd be able to count 10 items that fast.

Pirahã language doesn't even have numerals - that's an extreme case, but there quite a few languages where people stop counting beyond certain small number and just say "a lot". Same people though don't have issues lying to one another. Let that sink in for a while - fully grown-ass adults, fully capable of functioning in their society, not capable of counting one-two-three because the concept is beyond them.

What I'm trying to say is that all of those "requires theory of mind" statements are probably true but completely irrelevant because humans (and LLMs) have "hardware acceleration" of whatever it takes to lie, meanwhile counting is an abstract idea that requires to use the brain in a way it didn't evolve to be used. Similarly, LLMs cannot count if they aren't connected to a math engine - not because they're stupid, but because counting is really difficult.