←back to thread

GPT-5.2

(openai.com)
1053 points atgctg | 1 comments | 11 Dec 25 18:04 UTC | HN request time: 0.201s | source
Show context
josalhor ◴[11 Dec 25 18:24 UTC] No.46235005[source]▶
From GPT 5.1 Thinking:

ARC AGI v2: 17.6% -> 52.9%

SWE Verified: 76.3% -> 80%

That's pretty good!

replies(7): >>46235062 #>>46235070 #>>46235153 #>>46235160 #>>46235180 #>>46235421 #>>46236242 #
verdverm ◴[11 Dec 25 18:28 UTC] No.46235062[source]▶
We're also in benchmark saturation territory. I heard it speculated that Anthropic emphasizes benchmarks less in their publications because internally they don't care about them nearly as much as making a model that works well on the day-to-day
replies(5): >>46235126 #>>46235266 #>>46235466 #>>46235492 #>>46235583 #
Mistletoe ◴[11 Dec 25 18:43 UTC] No.46235266[source]▶
How do you measure whether it works better day to day without benchmarks?
replies(3): >>46235305 #>>46235348 #>>46235398 #
1. bulbar ◴[11 Dec 25 18:48 UTC] No.46235348[source]▶
Manually labeling answers maybe? There exist a lot of infrastructure built around and as it's heavily used for 2 decades and it's relatively cheap.

That's still benchmarking of course, but not utilizing any of the well known / public ones.