←back to thread

Something weird is happening with LLMs and chess

(dynomight.substack.com)
688 points crescit_eundo | 3 comments | 14 Nov 24 17:05 UTC | HN request time: 0.781s | source
Show context
azeirah ◴[14 Nov 24 22:43 UTC] No.42141993[source]▶
Maybe I'm really stupid... but perhaps if we want really intelligent models we need to stop tokenizing at all? We're literally limiting what a model can see and how it percieves the world by limiting the structure of the information streams that come into the model from the very beginning.

I know working with raw bits or bytes is slower, but it should be relatively cheap and easy to at least falsify this hypothesis that many huge issues might be due to tokenization problems but... yeah.

Surprised I don't see more research into radicaly different tokenization.

replies(14): >>42142033 #>>42142384 #>>42143197 #>>42143338 #>>42143381 #>>42144059 #>>42144207 #>>42144582 #>>42144600 #>>42145725 #>>42146419 #>>42146444 #>>42149355 #>>42151016 #
1. PittleyDunkin ◴[15 Nov 24 12:48 UTC] No.42146419[source]▶
A byte is itself sort of a token. So is a bit. It makes more sense to use more tokenizers in parallel than it does to try and invent an entirely new way of seeing the world.

Anyway humans have to tokenize, too. We don't perceive the world as a continuous blob either.

replies(1): >>42148069 #
2. samatman ◴[15 Nov 24 16:00 UTC] No.42148069[source]▶
I would say that "humans have to tokenize" is almost precisely the opposite of how human intelligence works.

We build layered, non-nested gestalts out of real time analog inputs. As a small example, the meaning of a sentence said with the same precise rhythm and intonation can be meaningfully changed by a gesture made while saying it. That can't be tokenized, and that isn't what's happening.

replies(1): >>42149700 #
3. PittleyDunkin ◴[15 Nov 24 18:50 UTC] No.42149700[source]▶
What is a gestalt if not a token (or a token representing collections of other tokens)? It seems more reasonable (to me) to conclude that we have multiple contradictory tokenizers that we select from rather than to reject the concept entirely.

> That can't be tokenized

Oh ye of little imagination.