BERT is just a single text diffusion step

This seems likely just from the simple fact that they can reliably generate contextually correct sentences in e.g. German Imperfekt.

It's actually true on many levels, if you think about is needed for generating syntactically and grammatically correct sentences, coherent text and working code.

I don't think you're wrong but I don't think your logic holds up here. If you have a literal translation like:

I have a hot dog _____

The word in the blank is not necessarily determined when the sentenced is started. Several verbs fit at the end and the LLM doesn't need to know which it's going to pick when it starts. Each word narrows down the possibilities:

I - Trillions Have - Billions a - millions hot - thousands dog - dozens _____ - Could be eaten, cooked, thrown, whatever.

If it chooses cooked at this point that doesn't necessarily mean that the LLM was going to do that when it chose "I" or "have"

That's why I hedged with "seems likely" and added "in context." If this is in the middle of a paragraph, then there are many fewer options to fit in the blank from the very start.

And, to pick an example from the research, being able to generate output that rhymes. In fact, it's hard to see how you would produce anything that would be considered coherent text without some degree of planning ahead at some level of abstraction. If it was truly one token at a time without any regard for what comes next it would constantly 'paint itself into a corner' and be forced to produce nonsense (which, it seems, does still happen sometimes, but without any planning it would occur constantly).

Just generating syntactically and grammatically correct sentences doesn't need much lookahead; prefixes to sentences that cannot be properly completed are going to be extremely unlikely to be generated.