←back to thread

A Research Preview of Codex

(openai.com)
420 points meetpateltech | 1 comments | 16 May 25 15:02 UTC | HN request time: 0.205s | source
Show context
ofirpress ◴[16 May 25 17:51 UTC] No.44008115[source]▶
[I'm one of the co-creators of SWE-bench] The team managed to improve on the already very strong o3 results on SWE-bench, but it's interesting that we're just seeing an improvement of a few percentage points. I wonder if getting to 85% from 75% on Verified is going to take as long as it took to get from 20% to 75%.
replies(2): >>44008209 #>>44009418 #
Snuggly73 ◴[16 May 25 18:00 UTC] No.44008209[source]▶
I can be completely off base, but it feels to me like benchmaxxing is going on with swe-bench.

Look at the results from multi swe bench - https://multi-swe-bench.github.io/#/

swe polybench - https://amazon-science.github.io/SWE-PolyBench/

Kotlin bench - https://firebender.com/leaderboard

replies(2): >>44010749 #>>44011138 #
1. Bjorkbat ◴[16 May 25 23:36 UTC] No.44010749[source]▶
I kind of had the feeling LLMs would be better at Python vs other languages, but wow, the difference on Multi SWE is pretty crazy.