←back to thread

Ask HN: Any insider takes on Yann LeCun's push against current architectures?

385 points vessenes | 4 comments | 10 Mar 25 19:41 UTC | HN request time: 0s | source

So, Lecun has been quite public saying that he believes LLMs will never fix hallucinations because, essentially, the token choice method at each step leads to runaway errors -- these can't be damped mathematically.

In exchange, he offers the idea that we should have something that is an 'energy minimization' architecture; as I understand it, this would have a concept of the 'energy' of an entire response, and training would try and minimize that.

Which is to say, I don't fully understand this. That said, I'm curious to hear what ML researchers think about Lecun's take, and if there's any engineering done around it. I can't find much after the release of ijepa from his group.

Show context
inimino ◴[14 Mar 25 20:47 UTC] No.43367126[source]▶
I have a paper coming up that I modestly hope will clarify some of this.

The short answer should be that it's obvious LLM training and inference are both ridiculously inefficient and biologically implausible, and therefore there has to be some big optimization wins still on the table.

replies(5): >>43367169 #>>43367233 #>>43367463 #>>43367776 #>>43367860 #
jedberg ◴[14 Mar 25 20:50 UTC] No.43367169[source]▶
> and biologically implausible

I really like this approach. Showing that we must be doing it wrong because our brains are more efficient and we aren't doing it like our brains.

Is this a common thing in ML papers or something you came up with?

replies(3): >>43367186 #>>43367478 #>>43368146 #
esafak ◴[14 Mar 25 20:52 UTC] No.43367186[source]▶
Evolution does not need to converge on the optimum solution.

Have you heard of https://en.wikipedia.org/wiki/Bio-inspired_computing ?

replies(2): >>43367202 #>>43367214 #
parsimo2010 ◴[14 Mar 25 20:56 UTC] No.43367214[source]▶
I don't think GP was implying that brains are the optimum solution. I think you can interpret GP's comments like this- if our brains are more efficient than LLMs, then clearly LLMs aren't optimally efficient. We have at least one data point showing that better efficiency is possible, even if we don't know what the optimal approach is.
replies(1): >>43367256 #
1. esafak ◴[14 Mar 25 21:00 UTC] No.43367256[source]▶
I agree. Spiking neural networks are usually mentioned in this context, but there is no hardware ecosystem behind them that can compete with Nvidia and CUDA.
replies(1): >>43367517 #
2. leereeves ◴[14 Mar 25 21:31 UTC] No.43367517[source]▶
Investments in AI are now counting by billions of dollars. Would that be enough to create an initial ecosystem for a new architecture?
replies(2): >>43367704 #>>43367775 #
3. esafak ◴[14 Mar 25 21:56 UTC] No.43367704[source]▶
Nvidia has a big lead, and hardware is capital intensive. I guess an alternative would make sense in the battery-powered regime, like robotics, where Nvidia's power hungry machines are at a disadvantage. This is how ARM took on Intel.
4. vlovich123 ◴[14 Mar 25 22:06 UTC] No.43367775[source]▶
A new HW architecture for an unproven SW architecture is never going to happen. The SW needs to start working initially and demonstrate better performance. Of course, as with the original deep neural net stuff, it took computers getting sufficiently advanced to demonstrate this is possible. A different SW architecture would have to be so much more efficient to work. Moreover, HW and SW evolve in tandem - HW takes existing SW and tries to optimize it (e.g. by adding an abstraction layer) or SW tries to leverage existing HW to run a new architecture faster. Coming up with a new HW/SW combo seems unlikely given the cost of bringing HW to market. If AI speedup of HW ever delivers like Jeff Dean expects, then the cost of prototyping might come down enough to try to make these kinds of bets.