←back to thread

Reasoning models reason well, until they don't

(arxiv.org)
214 points optimalsolver | 1 comments | 31 Oct 25 09:23 UTC | HN request time: 0s | source
Show context
equinox_nl ◴[31 Oct 25 09:48 UTC] No.45770131[source]▶
But I also fail catastrophically once a reasoning problem exceeds modest complexity.
replies(4): >>45770215 #>>45770281 #>>45770402 #>>45770506 #
monkeydust ◴[31 Oct 25 10:00 UTC] No.45770215[source]▶
But you recognise you are likely to fail and thus dont respond or redirect the problem to someone who has a greater likelihood of not failing.
replies(2): >>45770433 #>>45770440 #
antonvs ◴[31 Oct 25 10:30 UTC] No.45770433[source]▶
I’ve had models “redirect the problem to someone who has a greater likelihood of not failing”. Gemini in particular will do this when it runs into trouble.

I don’t find all these claims that models are somehow worse than humans in such areas convincing. Yes, they’re worse in some respects. But when you’re talking about things related to failures and accuracy, they’re mostly superhuman.

For example, how many humans can write hundred of lines of code (in seconds mind you) and regularly not have any syntax errors or bugs?

replies(2): >>45770627 #>>45772112 #
1. pessimizer ◴[31 Oct 25 14:05 UTC] No.45772112[source]▶
> I’ve had models “redirect the problem to someone who has a greater likelihood of not failing”. Gemini in particular will do this when it runs into trouble.

I have too, and I sense that this is something that has been engineered in rather than coming up naturally. I like it very much and they should do it a lot more often. They're allergic to "I can't figure this out" but hearing "I can't figure this out" gives me the alert to help it over the hump.

> But when you’re talking about things related to failures and accuracy, they’re mostly superhuman.

Only if you consider speed to failure and inaccuracy. They're very much subhuman in output, but you can make them retry a lot in a short time, and refine what you're asking them each time to avoid the mistakes they're repeatedly making. But that's you doing the work.