My 2.5 year old laptop can write Space Invaders in JavaScript now (GLM-4.5 Air)

Most likely its training data included countless Space Invaders in various programming languages.

and probably some synthetic data are generated copy of the games already on the dataset?

i have this feeling with LLM's generated react frontend, they all look the same

Last time somebody asked for a "premium camera app for iOS", and the model (re)generated Halide.

Models don't emit something they don't know. They remix and rewrite what they know. There's no invention, just recall...

> They remix and rewrite what they know. There's no invention, just recall...

If they only recalled they wouldn’t “hallucinate”. What’s a lie if not an invention? So clearly they can come up with data that they weren’t trained on, for better or worse.

Because internally, there isn't a difference between correctly "recalled" token and incorrectly (hallucinated).

Depends on the training? If there was eg RLHF then those connections are stronger and more likely; that's a difference (but not a category difference).