Getting 50% (SoTA) on Arc-AGI with GPT-4o

The Arc stuff just felt intuitively wrong as soon as I heard it. I don't find any of Chollet's critiques of LLMs to be convincing. It's almost as if he's being overly negative about them to make a point or something to push back against all the unbridled optimism. The problem is, the optimism really seems to be justified, and the rate of improvement of LLMs in the past 12 months has been nothing short of astonishing.

So it's not at all surprising to me to see Arc already being mostly solved using existing models, just with different prompting techniques and some tool usage. At some point, the naysayers about LLMs are going to have to confront the problem that, if they are right about LLMs not really thinking/understanding/being sentient, then a very large percentage of people living today are also not thinking/understanding/sentient!

> a very large percentage of people living today are also not thinking/understanding/sentient

This isn't that big a bullet to bite (https://www.lesswrong.com/posts/4AHXDwcGab5PhKhHT/humans-who... comes from well before ChatGPT's launch), and I myself am inclined to bite it. System 1 alone does not a general intelligence make, although the article is extremely interesting in asking the question "is System 1 plus Python enough for a general intelligence?". But it's not a very relevant philosophical point, because Chollet's position is consistent with humans being obsoleted and/or driven extinct whether or not the LLMs are "general intelligences".

His position is that training LLMs results in an ever-larger number of learned algorithms and no ability to construct new algorithms. This is consistent with the possibility that, after some threshold of size and training, the LLM has learned every algorithm it needs to supplant humans in (say) 99.9% of cases. (It would definitely be going out with a whimper rather than a bang, on that hypothesis, to be out-competed by something that _really is_ just a gigantic lookup table!)