←back to thread

My 2.5 year old laptop can write Space Invaders in JavaScript now (GLM-4.5 Air)

(simonwillison.net)
577 points simonw | 3 comments | 29 Jul 25 13:45 UTC | HN request time: 0s | source
Show context
AlexeyBrin ◴[29 Jul 25 14:02 UTC] No.44723521[source]▶
Most likely its training data included countless Space Invaders in various programming languages.
replies(6): >>44723664 #>>44723707 #>>44723945 #>>44724116 #>>44724439 #>>44724690 #
quantumHazer ◴[29 Jul 25 14:15 UTC] No.44723664[source]▶
and probably some synthetic data are generated copy of the games already on the dataset?

i have this feeling with LLM's generated react frontend, they all look the same

replies(4): >>44723867 #>>44724566 #>>44724902 #>>44731430 #
bayindirh ◴[29 Jul 25 14:29 UTC] No.44723867[source]▶
Last time somebody asked for a "premium camera app for iOS", and the model (re)generated Halide.

Models don't emit something they don't know. They remix and rewrite what they know. There's no invention, just recall...

replies(4): >>44724102 #>>44724181 #>>44724845 #>>44726775 #
mr_toad ◴[29 Jul 25 18:29 UTC] No.44726775[source]▶
> They remix and rewrite what they know. There's no invention, just recall...

If they only recalled they wouldn’t “hallucinate”. What’s a lie if not an invention? So clearly they can come up with data that they weren’t trained on, for better or worse.

replies(1): >>44727316 #
1. 0x457 ◴[29 Jul 25 19:26 UTC] No.44727316{3}[source]▶
Because internally, there isn't a difference between correctly "recalled" token and incorrectly (hallucinated).
replies(1): >>44734656 #
2. pbhjpbhj ◴[30 Jul 25 14:24 UTC] No.44734656[source]▶
Depends on the training? If there was eg RLHF then those connections are stronger and more likely; that's a difference (but not a category difference).
replies(1): >>44759348 #
3. 0x457 ◴[01 Aug 25 16:53 UTC] No.44759348[source]▶
Yes, but I thought we're talking about category difference.

Proper RLHF surely boosts "predicted next token until it couldn't" to feel more like "actually recalled".