←back to thread

BERT is just a single text diffusion step

(nathan.rs)
454 points nathan-barry | 1 comments | 20 Oct 25 14:31 UTC | HN request time: 0s | source
Show context
kibwen ◴[20 Oct 25 15:52 UTC] No.45645307[source]▶
To me, the diffusion-based approach "feels" more akin to whats going on in an animal brain than the token-at-a-time approach of the in-vogue LLMs. Speaking for myself, I don't generate words one a time based on previously spoken words; I start by having some fuzzy idea in my head and the challenge is in serializing it into language coherently.
replies(14): >>45645350 #>>45645383 #>>45645401 #>>45645402 #>>45645509 #>>45645523 #>>45645607 #>>45645665 #>>45645670 #>>45645891 #>>45645973 #>>45647491 #>>45648578 #>>45652892 #
1. aabhay ◴[20 Oct 25 15:57 UTC] No.45645383[source]▶
The fact that you’re cognitively aware is evidence that this is nowhere near diffusion. More like rumination or thinking tokens, if we absolutely had to find a present day LLM metaphor