←back to thread

BERT is just a single text diffusion step

(nathan.rs)
454 points nathan-barry | 1 comments | 20 Oct 25 14:31 UTC | HN request time: 0.001s | source
Show context
kibwen ◴[20 Oct 25 15:52 UTC] No.45645307[source]▶
To me, the diffusion-based approach "feels" more akin to whats going on in an animal brain than the token-at-a-time approach of the in-vogue LLMs. Speaking for myself, I don't generate words one a time based on previously spoken words; I start by having some fuzzy idea in my head and the challenge is in serializing it into language coherently.
replies(14): >>45645350 #>>45645383 #>>45645401 #>>45645402 #>>45645509 #>>45645523 #>>45645607 #>>45645665 #>>45645670 #>>45645891 #>>45645973 #>>45647491 #>>45648578 #>>45652892 #
1. EGreg ◴[20 Oct 25 16:07 UTC] No.45645509[source]▶
I feel completely the opposite way.

When you speak or do anything, you focus on what you’re going do next. Your next action. And at that moment you are relying on your recent memory, and things you have put in place while doing the overall activity (context).

In fact what’s actually missing from AI currently is simultaneous collaboration, like a group of people interacting — it is very 1 on 1 for now. Like human conversations.

Diffusion is like looking at a cloud and trying to find a pattern.

replies(1): >>45646015 #