Getting 50% (SoTA) on Arc-AGI with GPT-4o

> 50% accuracy on the public test set for ARC-AGI by having GPT-4o

Isn't the public test set public on github and therefore GPT-4o trained on it?

I keep seeing this comment all over the place. Just because something exists 1 time in the training data doesn't mean it can just regurgitate that. That's not how training works. An LLM is not a knowledge database.

And yet it doesn't rule out that it can't. See new york times lawsuit

In this case I don't think having seen the Arc set would help much in writing and selecting python scripts to solve the test cases. (Unless someone else has tried this approach before and _their_ results are in the training data.)

It will be good to see the private set results though.

From old pieces of articles that are quoted all over the internet? That's not surprising.

Public discussions of solutions to the public test set will presumably have somewhat similar analogies and/or embeddings to aspects of the python programs that solve them.

It could exist many times. People can fork and clone the repo. People are likely to copy the examples and share them online.

Yes, someone has tried https://iprc-dip.github.io/DARC/

That's still sufficient for both The Times and for it to be a potential problem in this case.