←back to thread

I think Yann Lecun was right about LLMs (but perhaps only by accident)

(substack.com)
129 points jxmorris12 | 1 comments | 21 Feb 25 18:23 UTC | HN request time: 0.224s | source
Show context
betula_ai ◴[21 Feb 25 18:50 UTC] No.43131365[source]▶
Thank you for this informative and thoughtful post. An interesting twist to the increasing error accumulation as autoregressive models generate more output, is the recent success of language diffusion models for predicting multiple tokens simultaneously. They have a remasking strategy at every step of the reviser process, that masks low confidence tokens. Regardless your observations perhaps still apply. https://arxiv.org/pdf/2502.09992
replies(1): >>43131724 #
1. jxmorris12 ◴[21 Feb 25 19:20 UTC] No.43131724[source]▶
Thanks for bringing this up! As far as I understand it current text diffusion models are limited to fairly short context windows. The idea of a text diffusion model continuously updating and revising a million-token-long chain-of-thought is pretty mind-boggling. I agree that these non-autoregressive models could potentially behave in completely different ways.

That said, I'm pretty sure we're a long way from building equally-competent diffusion-based base models, let alone reasoning models.

If anyone's interested in this topic, here are some more foundational papers to take a look at:

- Simple and Effective Masked Diffusion Language Models [2024] (https://arxiv.org/abs/2406.07524)

- Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution [2023] (https://arxiv.org/abs/2310.16834)

- Diffusion-LM Improves Controllable Text Generation [2022] (https://arxiv.org/abs/2205.14217)