I strongly believe that this will be a really important technique in the near future. The cost saving this might create is mouth watering.
replies(1):
I share the same belief, but regardless of cost. What excites me is the ability to "go both ways", edit previous tokens after others have been generated, using other signals as "guided generation", and so on. Next token prediction works for "stories", but diffusion matches better with "coding flows" (i.e. going back and forth, add something, come back, import something, edit something, and so on).
It would also be very interesting to see how applying this at different "abstraction layers" would work. Say you have one layer working on ctags, one working on files, and one working on "functions". And they all "talk" to each other, passing context and "re-diffusing" their respective layers after each change. No idea where the data for this would come, maybe from IDEs?
A lot of people these days are asking for structured output from LLMs so that a schema is followed. Even if you train on schema-following with a transformer, you're still just 'hoping' in the end that the generated json matches the schema.
I'm not a diffusion excerpt, but maybe there's a way to diffuse one value in the 'space' of numbers, and another value in the 'space' of all strings, as required by a schema:
{ "type": "object", "properties": { "amount": { "type": "number" }, "description": { "type": "string" } }, "required": ["amount", "description"] }
I'm not sure how far this could lead. Could you diffuse more complex schemas that generalize to a arbitrary syntax tree? E.g. diffuse some code in a programming language that is guaranteed to be type-safe?