←back to thread

The bitter lesson is coming for tokenization

(lucalp.dev)
296 points todsacerdoti | 1 comments | 24 Jun 25 14:14 UTC | HN request time: 0.303s | source
Show context
smeeth ◴[24 Jun 25 17:15 UTC] No.44368465[source]▶
The main limitation of tokenization is actually logical operations, including arithmetic. IIRC most of the poor performance of LLMs for math problems can be attributed to some very strange things that happen when you do math with tokens.

I'd like to see a math/logic bench appear for tokenization schemes that captures this. BPB/perplexity is fine, but its not everything.

replies(6): >>44368862 #>>44369438 #>>44371781 #>>44373480 #>>44374125 #>>44375446 #
calibas ◴[24 Jun 25 17:52 UTC] No.44368862[source]▶
It's a non-deterministic language model, shouldn't we expect mediocre performance in math? It seems like the wrong tool for the job...
replies(4): >>44368958 #>>44368999 #>>44369121 #>>44372463 #
1. CamperBob2 ◴[24 Jun 25 18:09 UTC] No.44368999[source]▶
We passed 'mediocre' a long time ago, but yes, it would be surprising if the same vocabulary representation is optimal for both verbal language and mathematical reasoning and computing.

To the extent we've already found that to be the case, it's perhaps the weirdest part of this whole "paradigm shift."