←back to thread

688 points crescit_eundo | 10 comments | | HN request time: 1.513s | source | bottom
Show context
azeirah ◴[] No.42141993[source]
Maybe I'm really stupid... but perhaps if we want really intelligent models we need to stop tokenizing at all? We're literally limiting what a model can see and how it percieves the world by limiting the structure of the information streams that come into the model from the very beginning.

I know working with raw bits or bytes is slower, but it should be relatively cheap and easy to at least falsify this hypothesis that many huge issues might be due to tokenization problems but... yeah.

Surprised I don't see more research into radicaly different tokenization.

replies(14): >>42142033 #>>42142384 #>>42143197 #>>42143338 #>>42143381 #>>42144059 #>>42144207 #>>42144582 #>>42144600 #>>42145725 #>>42146419 #>>42146444 #>>42149355 #>>42151016 #
numpad0 ◴[] No.42144207[source]
hot take: LLM tokens is kanji for AI, and just like kanji it works okay sometimes but fails miserably for the task of accurately representating English
replies(2): >>42148388 #>>42150181 #
umanwizard ◴[] No.42148388[source]
Why couldn’t Chinese characters accurately represent English? Japanese and Korean aren’t related to Chinese and still were written with Chinese characters (still are in the case of Japanese).

If England had been in the Chinese sphere of influence rather than the Roman one, English would presumably be written with Chinese characters too. The fact that it used an alphabet instead is a historical accident, not due to any grammatical property of the language.

replies(2): >>42150302 #>>42150512 #
stickfigure ◴[] No.42150302[source]
If I read you correctly, you're saying "the fact that the residents of England speak English instead of Chinese is a historical accident" and maybe you're right.

But the residents of England do in fact speak English, and English is a phonetic language, so there's an inherent impedance mismatch between Chinese characters and English language. I can make up words in English and write them down which don't necessarily have Chinese written equivalents (and probably, vice-versa?).

replies(1): >>42150670 #
1. umanwizard ◴[] No.42150670[source]
> If I read you correctly, you're saying "the fact that the residents of England speak English instead of Chinese is a historical accident" and maybe you're right.

That’s not what I mean at all. I mean even if spoken English were exactly the same as it is now, it could have been written with Chinese characters, and indeed would have been if England had been in the Chinese sphere of cultural influence when literacy developed there.

> English is a phonetic language

What does it mean to be a “phonetic language”? In what sense is English “more phonetic” than the Chinese languages?

> I can make up words in English and write them down which don’t necessarily have Chinese written equivalents

Of course. But if English were written with Chinese characters people would eventually agree on characters to write those words with, just like they did with all the native Japanese words that didn’t have Chinese equivalents but are nevertheless written with kanji.

Here is a famous article about how a Chinese-like writing system would work for English: https://www.zompist.com/yingzi/yingzi.htm

replies(2): >>42154464 #>>42155104 #
2. stickfigure ◴[] No.42154464[source]
> In what sense is English “more phonetic” than the Chinese languages?

Written English vs written Chinese.

How would you write, in Chinese, the words thingamajibber, gizmosity, or half the things that come out of AvE's mouth? These words have subtle, humorous, and entertaining meanings by way of twisting the sounds of other existing words. Shakespeare was a master of this kind of wordplay and invented a surprising number of words we use today.

I'm not saying you can't have the same phenomenon in spoken Chinese. But how do you write it down without a phonetic alphabet? And if you can't write it down, how do you share it to a wide audience?

replies(1): >>42154867 #
3. umanwizard ◴[] No.42154867[source]
> How would you write, in Chinese, the words thingamajibber, gizmosity, or half the things that come out of AvE's mouth?

With Chinese characters, of course. Why wouldn’t you be able to?

In English “thing”, “a”, and “ma” are already words, and “jibber” would presumably be the first character in “gibberish”. So you could write that made-up word by combining those four characters.

> But how do you write it down without a phonetic alphabet?

In general to write a newly coined word you would repurpose characters that sound the same as the newly coined word.

Every syllable that can possibly be uttered according to mandarin phonology is represented by some character (usually many), so this is always possible.

---

Regardless, to reiterate the original point: I'm not claiming Chinese characters are better or more flexible than alphabetic writing. They're not. I'm simply claiming that there's no inherent property of Japanese that makes it more amenable to representation with Chinese characters than English is (other than the fact that a lot of its vocabulary comes from Chinese, but that's not a real counterpoint given that there is lots of native, non-Chinese-derived vocabulary that's still written with kanji).

It would be possible to write Japanese entirely in the Latin alphabet, or English entirely with some system similar to Chinese characters, with minimal to no change to the structure of the language.

replies(2): >>42155215 #>>42162019 #
4. numpad0 ◴[] No.42155104[source]
"Donald Trump" in CJK, taken from Wikipedia page URL and as I hear it - each are close enough[1] and natural enough in each respective languages but none of it are particularly useful for counting R in strawberry:

  C: 唐納·川普, "Thangnar Changpooh"  
  J: ドナルド・トランプ, "Donaludo Toranpu"  
  K: 도널드 트럼프, "D'neldeh Tlempeuh"  
> What does it mean to be a “phonetic language”?

Means the script is intended to record pronunciation rather than intention, e.g. it's easy to see how "cow" is intended to be pronounced but it's not necessarily clear what a cow is; ideographic script on the other hand focuses on meaning, e.g. "魚" is supposed to look like a fish but pronunciation varies from "yueh", "sakana", "awe", etc.

1: I tried looking up other notable figures, but thought this person having entertainment background tends to illustrate the point more clearly

5. numpad0 ◴[] No.42155215{3}[source]
> I'm simply claiming that there's no inherent property of Japanese that makes it more amenable to representation with Chinese characters than English is

what? No, anything but IPA(only technically) and that language's native writings work for pronunciations. Hiragana, Hangul, or Chữ Quốc Ngữ, would not exist otherwise.

e: would _not_ exist

replies(1): >>42155340 #
6. umanwizard ◴[] No.42155340{4}[source]
Then why are both English and Latin represented with Latin characters despite having a completely different phoneme inventory?
replies(1): >>42156141 #
7. numpad0 ◴[] No.42156141{5}[source]
Because one is distant ancestor of the other...? It never adopted writing system from outside. The written and spoken systems co-evolved from a clean slate.
replies(1): >>42158827 #
8. umanwizard ◴[] No.42158827{6}[source]
That’s not true. English is not a descendant of Latin, and the Latin alphabet was adopted from the outside, replacing Anglo-Saxon runes (also called the Futhorc script).

Just like kanji are not native to Japanese.

9. stickfigure ◴[] No.42162019{3}[source]
> In English “thing”, “a”, and “ma” are already words, and “jibber” would presumably be the first character in “gibberish”. So you could write that made-up word by combining those four characters.

Nonsense. There is zero chance in hell that if you combine the pictographs for "thing", "a", "ma", and "gibberish", that someone reading that is going to reproduce the sound thingamajibber. It just does not work. The meme does not replicate.

There may be other virtues of pictographic written language, but reproducing sounds is not one of them. And - as any Shakespeare fan will tell you - tweaking the sounds of English cleverly is rather important. If you can't reproduce this behavior, you're losing something in translation. So to speak.

replies(1): >>42162107 #
10. umanwizard ◴[] No.42162107{4}[source]
Chinese characters aren't pictographs, so whether English could be written with pictographs is irrelevant to this discussion.

Each Chinese character represents a syllable (in Chinese languages) or a small set of possible sequences of syllables (in Japanese).

And yes, in Chinese languages, new words are created from characters that sound like the parts of the new word, all the time.