←back to thread

The bitter lesson is coming for tokenization

(lucalp.dev)
296 points todsacerdoti | 4 comments | 24 Jun 25 14:14 UTC | HN request time: 0.225s | source
Show context
marcosdumay ◴[24 Jun 25 16:31 UTC] No.44367966[source]▶
Yeah, make the network deeper.

When all you have is a hammer... It makes a lot of sense that a transformation layer that makes the tokens more semantically relevant will help optimize the entire network after it and increase the effective size of your context window. And one of the main immediate obstacle stopping those models from being intelligent is context window size.

On the other hand, the current models already cost something on the line of the median country GDP to train, and they are nowhere close to that in value. The saying that "if brute force didn't solve your problem, you didn't apply enough force" is intended to be listened as a joke.

replies(4): >>44368591 #>>44368640 #>>44374381 #>>44377110 #
jagraff ◴[24 Jun 25 17:27 UTC] No.44368591[source]▶
I think the median country GDP is something like $100 Billion

https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(PPP)

Models are expensive, but they're not that expensive.

replies(5): >>44368728 #>>44368795 #>>44368858 #>>44369293 #>>44371000 #
marcosdumay ◴[24 Jun 25 21:00 UTC] No.44371000[source]▶
$100 billion is the best estimate around of how much OpenAI took in investment to build ChatGPT.
replies(2): >>44372513 #>>44372608 #
1. Eisenstein ◴[25 Jun 25 00:43 UTC] No.44372608[source]▶
A top of the line consumer desktop, the Mac Pro, costs $7000. The commonly acknowledged first non-mechanical computer, the ENIAC, cost $400,000, which adjusted for inflation is $6,594,153 (see note). Will AI models follow the same pricing trajectory? Probably not but they no longer cost even close to $100 billion.

Note: 1946 CPI = 19.5, 2025 CPI = 321.465 which makes for an increase of 16.49.

replies(2): >>44374973 #>>44377745 #
2. ayewo ◴[25 Jun 25 08:38 UTC] No.44374973[source]▶
It seems all you did was use this formula:

CPI{2025} / CPI{1946} * Price{1946} = Price{2025}

to obtain the price adjusted for inflation?

That is the only way I was able to arrive at the same number you got: $6,594,153.846. TIL.

3. marcosdumay ◴[25 Jun 25 14:28 UTC] No.44377745[source]▶
> Will AI models follow the same pricing trajectory?

If the article is correct, and this is the best way to make them, their price will explode.

replies(1): >>44387046 #
4. mr_toad ◴[26 Jun 25 13:05 UTC] No.44387046[source]▶
Step 1: Spend hundreds of billions on AI

Step 2: ?

Step 3: Profit.

It’s not enough to have the biggest model, or the best model per dollar spent, you still need to figure out how to make money with it. It’s not clear than vastly increased expenditure will produce a good ROI.