←back to thread

Getting 50% (SoTA) on Arc-AGI with GPT-4o

(redwoodresearch.substack.com)
394 points tomduncalf | 3 comments | | HN request time: 0.215s | source
Show context
rgbrgb ◴[] No.40712154[source]
> 50% accuracy on the public test set for ARC-AGI by having GPT-4o

Isn't the public test set public on github and therefore GPT-4o trained on it?

replies(2): >>40712401 #>>40712472 #
1. daemonologist ◴[] No.40712472[source]
In this case I don't think having seen the Arc set would help much in writing and selecting python scripts to solve the test cases. (Unless someone else has tried this approach before and _their_ results are in the training data.)

It will be good to see the private set results though.

replies(2): >>40712793 #>>40713721 #
2. cma ◴[] No.40712793[source]
Public discussions of solutions to the public test set will presumably have somewhat similar analogies and/or embeddings to aspects of the python programs that solve them.
3. Truth_In_Lies ◴[] No.40713721[source]
Yes, someone has tried https://iprc-dip.github.io/DARC/