(nathan.rs)

454 points nathan-barry | 1 comments | 20 Oct 25 14:31 UTC | HN request time: 0.247s | source

Show context

bonoboTP ◴[20 Oct 25 21:23 UTC] No.45649556[source]▶

It feels like it would make more sense to allow the model to do Levenshtein-like edits instead of just masking and filling in the masked tokens. It seems that intuitively it's really hard in this diffusion setup to just swap one word with a longer but better synonym towards the end, because there's no way to shift everything to the right afterwards.

replies(1): >>45649953 #

1. lucidrains ◴[20 Oct 25 21:58 UTC] No.45649953[source]▶

>>45649556 #

there has been some movement on that front, in the form of adding expand / delete tokens! https://hkunlp.github.io/blog/2025/dreamon/

↑

BERT is just a single text diffusion step