←back to thread

The bitter lesson is coming for tokenization

(lucalp.dev)
296 points todsacerdoti | 1 comments | 24 Jun 25 14:14 UTC | HN request time: 0.21s | source
Show context
smeeth ◴[24 Jun 25 17:15 UTC] No.44368465[source]▶
The main limitation of tokenization is actually logical operations, including arithmetic. IIRC most of the poor performance of LLMs for math problems can be attributed to some very strange things that happen when you do math with tokens.

I'd like to see a math/logic bench appear for tokenization schemes that captures this. BPB/perplexity is fine, but its not everything.

replies(6): >>44368862 #>>44369438 #>>44371781 #>>44373480 #>>44374125 #>>44375446 #
1. vendiddy ◴[25 Jun 25 06:18 UTC] No.44374125[source]▶
Do LLMs need to be good at math with the same approach?

To draw an a analogy, we've got our human brain specialized.

Why not implement a part of the AI brain that's not neural nets, but instead circuitry specialized to math?

Maybe a dumb question since I'm a layperson!