←back to thread

385 points vessenes | 1 comments | | HN request time: 0.241s | source

So, Lecun has been quite public saying that he believes LLMs will never fix hallucinations because, essentially, the token choice method at each step leads to runaway errors -- these can't be damped mathematically.

In exchange, he offers the idea that we should have something that is an 'energy minimization' architecture; as I understand it, this would have a concept of the 'energy' of an entire response, and training would try and minimize that.

Which is to say, I don't fully understand this. That said, I'm curious to hear what ML researchers think about Lecun's take, and if there's any engineering done around it. I can't find much after the release of ijepa from his group.

1. hansvm ◴[] No.43368709[source]
One small point: Token selection at each step is fine (and required if you want to be able to additively/linearly/independently handle losses). The problem here is the high inaccuracies in each token (or, rather, their distributions). If you use more time and space to generate the token then those errors go down. If using more time and space cannot suffice then, by construction, energy minimization models and any other solution you can think of also can't reduce the errors far enough.