←back to thread

BERT is just a single text diffusion step

(nathan.rs)
454 points nathan-barry | 1 comments | 20 Oct 25 14:31 UTC | HN request time: 0.355s | source
Show context
kibwen ◴[20 Oct 25 15:52 UTC] No.45645307[source]▶
To me, the diffusion-based approach "feels" more akin to whats going on in an animal brain than the token-at-a-time approach of the in-vogue LLMs. Speaking for myself, I don't generate words one a time based on previously spoken words; I start by having some fuzzy idea in my head and the challenge is in serializing it into language coherently.
replies(14): >>45645350 #>>45645383 #>>45645401 #>>45645402 #>>45645509 #>>45645523 #>>45645607 #>>45645665 #>>45645670 #>>45645891 #>>45645973 #>>45647491 #>>45648578 #>>45652892 #
cube2222 ◴[20 Oct 25 15:55 UTC] No.45645350[source]▶
I will very often write a message on slack, only to then edit it 5 times… Now I always feel like a diffusion model when I do that.
replies(1): >>45647460 #
1. djmips ◴[20 Oct 25 18:35 UTC] No.45647460[source]▶
Coding feels like that to me as well.