←back to thread

Getting 50% (SoTA) on Arc-AGI with GPT-4o

(redwoodresearch.substack.com)
394 points tomduncalf | 1 comments | 17 Jun 24 21:51 UTC | HN request time: 0.486s | source
Show context
rgbrgb ◴[17 Jun 24 22:58 UTC] No.40712154[source]▶
> 50% accuracy on the public test set for ARC-AGI by having GPT-4o

Isn't the public test set public on github and therefore GPT-4o trained on it?

replies(2): >>40712401 #>>40712472 #
bongodongobob ◴[17 Jun 24 23:27 UTC] No.40712401[source]▶
I keep seeing this comment all over the place. Just because something exists 1 time in the training data doesn't mean it can just regurgitate that. That's not how training works. An LLM is not a knowledge database.
replies(2): >>40712453 #>>40713177 #
1. spencerchubb ◴[18 Jun 24 01:30 UTC] No.40713177[source]▶
It could exist many times. People can fork and clone the repo. People are likely to copy the examples and share them online.