←back to thread

Something weird is happening with LLMs and chess

(dynomight.substack.com)
688 points crescit_eundo | 8 comments | 14 Nov 24 17:05 UTC | HN request time: 1.831s | source | bottom
Show context
azeirah ◴[14 Nov 24 22:43 UTC] No.42141993[source]▶
Maybe I'm really stupid... but perhaps if we want really intelligent models we need to stop tokenizing at all? We're literally limiting what a model can see and how it percieves the world by limiting the structure of the information streams that come into the model from the very beginning.

I know working with raw bits or bytes is slower, but it should be relatively cheap and easy to at least falsify this hypothesis that many huge issues might be due to tokenization problems but... yeah.

Surprised I don't see more research into radicaly different tokenization.

replies(14): >>42142033 #>>42142384 #>>42143197 #>>42143338 #>>42143381 #>>42144059 #>>42144207 #>>42144582 #>>42144600 #>>42145725 #>>42146419 #>>42146444 #>>42149355 #>>42151016 #
numpad0 ◴[15 Nov 24 05:34 UTC] No.42144207[source]▶
hot take: LLM tokens is kanji for AI, and just like kanji it works okay sometimes but fails miserably for the task of accurately representating English
replies(2): >>42148388 #>>42150181 #
umanwizard ◴[15 Nov 24 16:28 UTC] No.42148388[source]▶
Why couldn’t Chinese characters accurately represent English? Japanese and Korean aren’t related to Chinese and still were written with Chinese characters (still are in the case of Japanese).

If England had been in the Chinese sphere of influence rather than the Roman one, English would presumably be written with Chinese characters too. The fact that it used an alphabet instead is a historical accident, not due to any grammatical property of the language.

replies(2): >>42150302 #>>42150512 #
stickfigure ◴[15 Nov 24 19:58 UTC] No.42150302[source]▶
If I read you correctly, you're saying "the fact that the residents of England speak English instead of Chinese is a historical accident" and maybe you're right.

But the residents of England do in fact speak English, and English is a phonetic language, so there's an inherent impedance mismatch between Chinese characters and English language. I can make up words in English and write them down which don't necessarily have Chinese written equivalents (and probably, vice-versa?).

replies(1): >>42150670 #
umanwizard ◴[15 Nov 24 20:34 UTC] No.42150670[source]▶
> If I read you correctly, you're saying "the fact that the residents of England speak English instead of Chinese is a historical accident" and maybe you're right.

That’s not what I mean at all. I mean even if spoken English were exactly the same as it is now, it could have been written with Chinese characters, and indeed would have been if England had been in the Chinese sphere of cultural influence when literacy developed there.

> English is a phonetic language

What does it mean to be a “phonetic language”? In what sense is English “more phonetic” than the Chinese languages?

> I can make up words in English and write them down which don’t necessarily have Chinese written equivalents

Of course. But if English were written with Chinese characters people would eventually agree on characters to write those words with, just like they did with all the native Japanese words that didn’t have Chinese equivalents but are nevertheless written with kanji.

Here is a famous article about how a Chinese-like writing system would work for English: https://www.zompist.com/yingzi/yingzi.htm

replies(2): >>42154464 #>>42155104 #
1. stickfigure ◴[16 Nov 24 05:10 UTC] No.42154464[source]▶
> In what sense is English “more phonetic” than the Chinese languages?

Written English vs written Chinese.

How would you write, in Chinese, the words thingamajibber, gizmosity, or half the things that come out of AvE's mouth? These words have subtle, humorous, and entertaining meanings by way of twisting the sounds of other existing words. Shakespeare was a master of this kind of wordplay and invented a surprising number of words we use today.

I'm not saying you can't have the same phenomenon in spoken Chinese. But how do you write it down without a phonetic alphabet? And if you can't write it down, how do you share it to a wide audience?

replies(1): >>42154867 #
2. umanwizard ◴[16 Nov 24 06:39 UTC] No.42154867[source]▶
> How would you write, in Chinese, the words thingamajibber, gizmosity, or half the things that come out of AvE's mouth?

With Chinese characters, of course. Why wouldn’t you be able to?

In English “thing”, “a”, and “ma” are already words, and “jibber” would presumably be the first character in “gibberish”. So you could write that made-up word by combining those four characters.

> But how do you write it down without a phonetic alphabet?

In general to write a newly coined word you would repurpose characters that sound the same as the newly coined word.

Every syllable that can possibly be uttered according to mandarin phonology is represented by some character (usually many), so this is always possible.

---

Regardless, to reiterate the original point: I'm not claiming Chinese characters are better or more flexible than alphabetic writing. They're not. I'm simply claiming that there's no inherent property of Japanese that makes it more amenable to representation with Chinese characters than English is (other than the fact that a lot of its vocabulary comes from Chinese, but that's not a real counterpoint given that there is lots of native, non-Chinese-derived vocabulary that's still written with kanji).

It would be possible to write Japanese entirely in the Latin alphabet, or English entirely with some system similar to Chinese characters, with minimal to no change to the structure of the language.

replies(2): >>42155215 #>>42162019 #
3. numpad0 ◴[16 Nov 24 08:14 UTC] No.42155215[source]▶
> I'm simply claiming that there's no inherent property of Japanese that makes it more amenable to representation with Chinese characters than English is

what? No, anything but IPA(only technically) and that language's native writings work for pronunciations. Hiragana, Hangul, or Chữ Quốc Ngữ, would not exist otherwise.

e: would _not_ exist

replies(1): >>42155340 #
4. umanwizard ◴[16 Nov 24 08:51 UTC] No.42155340{3}[source]▶
Then why are both English and Latin represented with Latin characters despite having a completely different phoneme inventory?
replies(1): >>42156141 #
5. numpad0 ◴[16 Nov 24 12:47 UTC] No.42156141{4}[source]▶
Because one is distant ancestor of the other...? It never adopted writing system from outside. The written and spoken systems co-evolved from a clean slate.
replies(1): >>42158827 #
6. umanwizard ◴[16 Nov 24 20:03 UTC] No.42158827{5}[source]▶
That’s not true. English is not a descendant of Latin, and the Latin alphabet was adopted from the outside, replacing Anglo-Saxon runes (also called the Futhorc script).

Just like kanji are not native to Japanese.

7. stickfigure ◴[17 Nov 24 04:45 UTC] No.42162019[source]▶
> In English “thing”, “a”, and “ma” are already words, and “jibber” would presumably be the first character in “gibberish”. So you could write that made-up word by combining those four characters.

Nonsense. There is zero chance in hell that if you combine the pictographs for "thing", "a", "ma", and "gibberish", that someone reading that is going to reproduce the sound thingamajibber. It just does not work. The meme does not replicate.

There may be other virtues of pictographic written language, but reproducing sounds is not one of them. And - as any Shakespeare fan will tell you - tweaking the sounds of English cleverly is rather important. If you can't reproduce this behavior, you're losing something in translation. So to speak.

replies(1): >>42162107 #
8. umanwizard ◴[17 Nov 24 05:04 UTC] No.42162107{3}[source]▶
Chinese characters aren't pictographs, so whether English could be written with pictographs is irrelevant to this discussion.

Each Chinese character represents a syllable (in Chinese languages) or a small set of possible sequences of syllables (in Japanese).

And yes, in Chinese languages, new words are created from characters that sound like the parts of the new word, all the time.