←back to thread

385 points vessenes | 1 comments | | HN request time: 0.21s | source

So, Lecun has been quite public saying that he believes LLMs will never fix hallucinations because, essentially, the token choice method at each step leads to runaway errors -- these can't be damped mathematically.

In exchange, he offers the idea that we should have something that is an 'energy minimization' architecture; as I understand it, this would have a concept of the 'energy' of an entire response, and training would try and minimize that.

Which is to say, I don't fully understand this. That said, I'm curious to hear what ML researchers think about Lecun's take, and if there's any engineering done around it. I can't find much after the release of ijepa from his group.

Show context
eximius ◴[] No.43367519[source]
I believe that so long as weights are fixed at inference time, we'll be at a dead end.

Will Titans be sufficiently "neuroplastic" to escape that? Maybe, I'm not sure.

Ultimately, I think an architecture around "looping" where the model outputs are both some form of "self update" and "optional actionality" such that interacting with the model is more "sampling from a thought space" will be required.

replies(3): >>43367644 #>>43370757 #>>43372112 #
mft_ ◴[] No.43367644[source]
Very much this. I’ve been wondering why I’ve not seen it much discussed.
replies(2): >>43368224 #>>43369295 #
1. eximius ◴[] No.43369295[source]
Self updating requires learning to learn, which I'm not sure we know how to do.