←back to thread

Something weird is happening with LLMs and chess

(dynomight.substack.com)
688 points crescit_eundo | 3 comments | 14 Nov 24 17:05 UTC | HN request time: 0.671s | source
Show context
azeirah ◴[14 Nov 24 22:43 UTC] No.42141993[source]▶
Maybe I'm really stupid... but perhaps if we want really intelligent models we need to stop tokenizing at all? We're literally limiting what a model can see and how it percieves the world by limiting the structure of the information streams that come into the model from the very beginning.

I know working with raw bits or bytes is slower, but it should be relatively cheap and easy to at least falsify this hypothesis that many huge issues might be due to tokenization problems but... yeah.

Surprised I don't see more research into radicaly different tokenization.

replies(14): >>42142033 #>>42142384 #>>42143197 #>>42143338 #>>42143381 #>>42144059 #>>42144207 #>>42144582 #>>42144600 #>>42145725 #>>42146419 #>>42146444 #>>42149355 #>>42151016 #
1. jncfhnb ◴[15 Nov 24 01:43 UTC] No.42143197[source]▶
There’s a reason human brains have dedicated language handling. Tokenization is likely a solid strategy. The real thing here is that language is not a good way to encode all forms of knowledge
replies(1): >>42144149 #
2. joquarky ◴[15 Nov 24 05:19 UTC] No.42144149[source]▶
It's not even possible to encode all forms of knowledge.
replies(1): >>42145431 #
3. shaky-carrousel ◴[15 Nov 24 10:00 UTC] No.42145431[source]▶
I know a joke where half of the joke is whistling and half gesturing, and the punchline is whistling. The wording is basically just to say who the players are.