←back to thread

An LLM is a lossy encyclopedia

(simonwillison.net)
509 points tosh | 5 comments | | HN request time: 0.527s | source

(the referenced HN thread starts at https://news.ycombinator.com/item?id=45060519)
Show context
kgeist ◴[] No.45101806[source]
I think an LLM can be used as a kind of lossy encyclopedia, but equating it directly to one isn't entirely accurate. The human mind is also, in a sense, a lossy encyclopedia.

I prefer to think of LLMs as lossy predictors. If you think about it, natural "intelligence" itself can be understood as another type of predictor: you build a world model to anticipate what will happen next so you can plan your actions accordingly and survive.

In the real world, with countless fuzzy factors, no predictor can ever be perfectly lossless. The only real difference, for me, is that LLMs are lossier predictors than human minds (for now). That's all there is to it.

Whatever analogy you use, it comes down to the realization that there's always some lossiness involved, whether you frame it as an encyclopedia or not.

replies(6): >>45102030 #>>45102068 #>>45102070 #>>45102175 #>>45102917 #>>45103645 #
1. somewhereoutth ◴[] No.45102175[source]
> you build a world model

The foundational conceit (if you will) of LLMs is that they build a semantic (world) model to 'make sense' of their training. However it is much more likely that they are simply building a syntactic model in response to the training. As far as I know there is no evidence of a semantic model emerging.

replies(4): >>45102373 #>>45102575 #>>45102876 #>>45103249 #
2. jebarker ◴[] No.45102373[source]
Maybe I don’t have a precise enough definition of syntax and semantics, but it seems like it’s more than just syntactic since interchangeable tokens in the same syntax affect the semantics of the sentence. Or do you view completing a prompt such as “The president of the United States is?” as a syntax question?
3. ijk ◴[] No.45102575[source]
There's some evidence of valid relationships: you can build a map of Manhattan by asking about directions from each street corner and plotting the relations.

This is still entirely referential, but in a way that a human would see some relation to the actual thing, albeit in a somewhat weird and alien way.

4. ◴[] No.45102876[source]
5. IanCal ◴[] No.45103249[source]
Is this not addressed by othellogpt?