←back to thread

Ask HN: Any insider takes on Yann LeCun's push against current architectures?

385 points vessenes | 1 comments | 10 Mar 25 19:41 UTC | HN request time: 0.274s | source

So, Lecun has been quite public saying that he believes LLMs will never fix hallucinations because, essentially, the token choice method at each step leads to runaway errors -- these can't be damped mathematically.

In exchange, he offers the idea that we should have something that is an 'energy minimization' architecture; as I understand it, this would have a concept of the 'energy' of an entire response, and training would try and minimize that.

Which is to say, I don't fully understand this. That said, I'm curious to hear what ML researchers think about Lecun's take, and if there's any engineering done around it. I can't find much after the release of ijepa from his group.

Show context
giantg2 ◴[14 Mar 25 21:34 UTC] No.43367543[source]▶
I feel like some hallucinations aren't bad. Isn't that basically what a new idea is - a hallucination of what could be? The ability to come up with new things, even if they're sometimes wrong, can be useful and happen all the time with humans.
replies(2): >>43367585 #>>43372183 #
1. chriskanan ◴[15 Mar 25 12:43 UTC] No.43372183[source]▶
Hallucinations is a terrible term for this. We want models that can create new ideas and make up stories. The problem is that they will give false answers to questions with factual answers, and that they don't realize this.

In humans, this is known as confabulation, and it happens due to various forms of brain damage, especially with damage to orbitofrontal cortex (part of prefrontal cortex). David Rumelhart, who was the main person who came up with backpropagation in a paper co-authored with Geoff Hinton, actually got Pick's disease which specifically results in damage to prefrontal cortex and people with that disease exhibit a lot of the same problems we have with today's LLMs: