←back to thread

The bitter lesson is coming for tokenization

(lucalp.dev)
296 points todsacerdoti | 1 comments | 24 Jun 25 14:14 UTC | HN request time: 0.478s | source
Show context
citizenpaul ◴[24 Jun 25 18:14 UTC] No.44369060[source]▶
The best general argument I've heard against the bitter lesson is. If the bitter lesson is true? How come we spend so many million man hours a year of tweaking and optimizing software systems all day long? Surely its easier and cheaper to just buy a rack of servers.

Maybe if you have infinite compute you don't worry about software design. Meanwhile in the real world...

Not only that but where did all these compute optimized solutions come from? Oh yeah millions of man hours of optimizing and testing algorithmic solutions. So unless you are some head in the clouds tenured professor just keep on doing your optimizations and job as usual.

replies(3): >>44369284 #>>44370748 #>>44377181 #
1. awkward ◴[25 Jun 25 13:35 UTC] No.44377181[source]▶
There's some domains where the bitter lesson has big impacts on theory. The Peter Norvig vs Noam Chomsky debate on the merits of brute force compute finding a full and complete theory of language is an example. That's a case where the path of "get a ton of data and handle it statistically" competes with the path of "build a complete and abstract understanding of the domain." Lots of resources and lifetimes of work are decided by which path to take.

Agreed that overfitting the bitter lesson often leads slopping piles of compute and hardware at problems that could just be deterministic.