←back to thread

Getting 50% (SoTA) on Arc-AGI with GPT-4o

(redwoodresearch.substack.com)
394 points tomduncalf | 2 comments | 17 Jun 24 21:51 UTC | HN request time: 0.39s | source
Show context
atleastoptimal ◴[18 Jun 24 04:44 UTC] No.40714152[source]▶
I'll say what a lot of people seem to be denying. GPT-4 is an AGI, just a very bad one. Even GPT-1 was an AGI. There isn't a hard boundary between non AGI and AGI. A lot of people wish there was so they imagine absolutes regarding LLM's like "they cannot create anything new" or something like that. Just think: we consider humans a general intelligence, but obviously wouldn't consider an embryo or infant a general intelligence. So at what point does a human go from not generally intelligent to generally intelligent? And I don't mean an age or brain size, I mean suite of testable abilities.

Intelligence is an ability that is naturally gradual and emerges over many domains. It is a collection of tools via which general abstractive principles can be applied, not a singular universally applicable ability to think in abstractions. GPT-4, compared to a human, is a very very small brain trained for the single purpose of textual thinking with some image capabilities. Claiming that ARC is the absolute market of general intelligence fails to account for the big picture of what intelligence is.

replies(7): >>40714189 #>>40714191 #>>40714565 #>>40715248 #>>40715346 #>>40715384 #>>40716518 #
1. Tepix ◴[18 Jun 24 08:07 UTC] No.40715248[source]▶
The definition of AGI that i am familiar with is that it can do all (digital) tasks a human can do at the level of an average human.

As long as this level hasn't been achieved in all domains, it isn't AGI.

replies(1): >>40715391 #
2. lupusreal ◴[18 Jun 24 08:36 UTC] No.40715391[source]▶
This seems like a problematic standard. For one, it's very fuzzy. A human who is top 51% and one who's top 49% are very similar to each other and could probably swap places depending on how they're feeling that day; there's nothing fundamentally different going on in their heads. Even at the further ends of the scale, humans have essentially the same kind of brains and minds as each other, some more capable than others but still all belonging to the same category of thinking things. Your AGI definition bifurcates the human population into those that possess general intelligence and those who don't, but this seems hard to justify. At least, hard to justify when drawn there. If you put the line at profound mental retardation where a person can no longer function in society, that would make more sense. A slightly below average human may not be exceptional in any regard but I think they still possess what must be regarded as general intelligence.

Furthermore, you're counting cases where humans do things the computer cannot but ignoring cases where the computer does things humans cannot. For instance, I doubt any human alive, let alone average humans can give reasonable explanations for short snippets of computer code in as many languages as GPT-4o, or formulate poetry in as many styles on arbitrary topics, or rattle off esoteric trivia and opinions about obscure historic topics, .... I think you get the point. It has already surpassed average human abilities in many categories of intellectually challenging tasks, but with your definition if it fails at even one task an average human can do, then it lacks "general intelligence."

I suggest that your definition is one for "AHI" (Average Human Intelligence), not one for "AGI" (Artificial General Intelligence.)