(redwoodresearch.substack.com)

394 points tomduncalf | 2 comments | 17 Jun 24 21:51 UTC | HN request time: 0.401s | source

1. traject_ ◴[17 Jun 24 22:42 UTC] No.40712005[source]▶

We don't actually know if it is SOTA, the previous SOTA solution also got around the same on the evaluation set.

2. cma ◴[17 Jun 24 23:13 UTC] No.40712275[source]▶

Yeah and GPT4o was potentially trained on this test set and if the tried to hold it out it was still likely trained on discussions of the problems.

Getting 50% (SoTA) on Arc-AGI with GPT-4o