←back to thread

The bitter lesson is coming for tokenization

(lucalp.dev)
296 points todsacerdoti | 1 comments | 24 Jun 25 14:14 UTC | HN request time: 0.558s | source
Show context
blixt ◴[24 Jun 25 19:49 UTC] No.44370298[source]▶
I’m starting to think “The Bitter Lesson” is a clever sounding way to give shade to people that failed to nail it on their first attempt. Usually engineers build much more technology than they actually end up needing, then the extras shed off with time and experience (and often you end up building it again from scratch). It’s not clear to me that starting with “just build something that scales with compute” would get you closer to the perfect solution, even if as you get closer to it you do indeed make it possible to throw more compute at it.

That said the hand coded nature of tokenization certainly seems in dire need of a better solution, something that can be learned end to end. And It looks like we are getting closer with every iteration.

replies(3): >>44370492 #>>44370692 #>>44370704 #
QuesnayJr ◴[24 Jun 25 20:30 UTC] No.44370692[source]▶
I'm starting to think that half the commenters here don't actually know what "The Bitter Lesson" is. It's purely a statement about the history of AI research, in a very short essay by Rich Sutton: http://www.incompleteideas.net/IncIdeas/BitterLesson.html It's not some general statement about software engineering for all domains, but a very specific statement about AI applications. It's an observation that the previous generation's careful algorithmic work to solve an AI problem ends up being obsoleted by this generation's brute force approach using more computing power. It's something that's happened over and over again in AI, and has happened several times even since 2019 when Sutton wrote the essay.
replies(2): >>44370822 #>>44371113 #
1. tantalor ◴[24 Jun 25 20:43 UTC] No.44370822[source]▶
That essay is actually linked in the lead:

> As it's been pointed out countless times - if the trend of ML research could be summarised, it'd be the adherence to The Bitter Lesson - opt for general-purpose methods that leverage large amounts of compute and data over crafted methods by domain experts

But we're only 1 sentence in, and this is already a failure of science communication at several levels.

1. The sentence structure and grammar is simply horrible

2. This is condescending: "pointed out countless times" - has it?

3. The reference to Sutton's essay is oblique, easy to miss

4. Outside of AI circles, "Bitter Lesson" is not very well known. If you didn't already know about it, this doesn't help.

replies(1): >>44371290 #