←back to thread

The bitter lesson is coming for tokenization

(lucalp.dev)
296 points todsacerdoti | 1 comments | 24 Jun 25 14:14 UTC | HN request time: 0.283s | source
Show context
cheesecompiler ◴[24 Jun 25 15:32 UTC] No.44367317[source]▶
The reverse is possible too: throwing massive compute at a problem can mask the existence of a simpler, more general solution. General-purpose methods tend to win out over time—but how can we be sure they’re truly the most general if we commit so hard to one paradigm (e.g. LLMs) that we stop exploring the underlying structure?
replies(4): >>44367776 #>>44367991 #>>44368757 #>>44375546 #
1. falcor84 ◴[24 Jun 25 16:34 UTC] No.44367991[source]▶
The way I see this, from the explore-exploit point of view, it's pretty rational to put the vast majority of your effort into the one action that has shown itself to bring the most reward, while spending a small amount of effort exploring other ones. Then, if and when that one action is no longer as fruitful compared to the others, you switch more effort to exploring, now having obtained significant resources from that earlier exploration, to help you explore faster.