Ask HN: Any insider takes on Yann LeCun's push against current architectures?

I am an MLE not an expert. However, it is a fundamental problem that our current paradigm of training larger and larger LLMs cannot ever scale to the precision people require for many tasks. Even in the highly constrained realm of chess, an enormous neural net will be outclassed by a small program that can run on your phone.

https://arxiv.org/pdf/2402.04494

> Even in the highly constrained realm of chess, an enormous neural net will be outclassed by a small program that can run on your phone.

This is true also for the much bigger neural net that works in your brain, and even if you're the world champion of chess. Clearly your argument doesn't hold water.

Any chance that “reasoning” can fix this

For the sake of argument let’s say an artificial neural net is approximately the same as the brain. It sounds like you agree with me that smaller programs are both more efficient and more effective than a larger neural net. So you should also agree with me that those who say the only path to AGI is LLM maximalism are misguided.

It kind of depends. You can broadly call any kind of search “reasoning”. But search requires 1) enumerating your possible options and 2) assigning some value to those options. Real world problem solving makes both of those extremely difficult.

Unlike in chess, there’s a functionally infinite number of actions you can take in real life. So just argmax over possible actions is going to be hard.

Two, you have to have some value function of how good an action is in order to argmax. But many actions are impossible to know the value of in practice because of hidden information and the chaotic nature of the world (butterfly effect).

smaller programs are better than artificial or organic neural net for constrained problems like chess. But chess programs don't generalize to any other intelligence applications, like how organic neural nets do today.

> It sounds like you agree with me that smaller programs are both more efficient and more effective than a larger neural net.

At playing chess. (But also at doing sums and multiplications, yay!)

> So you should also agree with me that those who say the only path to AGI is LLM maximalism are misguided.

No. First of all, it's a claim you just made up. What we're talking about is people saying that LLMs are not the path to AGI- an entirely different claim.

Second, assuming there's any coherence to your argument, the fact that a small program can outclass an enormous NN is irrelevant to the question of whether the enormous NN is the right way to achieve AGI: we are "general intelligences" and we are defeated by the same chess program. Unless you mean that achieving the intelligence of the greatest geniuses that ever lived is still not enough.

Isn't something about alphago also involves "infinitely" many possible outcomes? Yet they cracked it, right?

Go is played on a 19x19 board. At the beginning of the game the first player has 361 possible moves. The second player then has 360 possible moves. There is always a finite and relatively “small” number of options.

I think you are thinking of the fact that it had to be approached in a different way than Minimax in chess because a brute force decision tree grows way too fast to perform well. So they had to learn models for actions and values.

In any case, Go is a perfect information game, which as I mentioned before, is not the same as problems in the real world.

Best in class chess program actually is a NN, just not a LLM.