←back to thread

Karpathy on DeepSeek-OCR paper: Are pixels better inputs to LLMs than text?

(twitter.com)
233 points JnBrymn | 8 comments | 21 Oct 25 17:43 UTC | HN request time: 1.561s | source | bottom
Show context
sabareesh ◴[22 Oct 25 22:18 UTC] No.45675879[source]▶
It might be that our current tokenization is inefficient compared to how well image pipeline does. Language already does lot of compression but there might be even better way to represent it in latent space
replies(3): >>45675953 #>>45676049 #>>45677115 #
ACCount37 ◴[22 Oct 25 22:26 UTC] No.45675953[source]▶
People in the industry know that tokenizers suck and there's room to do better. But actually doing it better? At scale? Now that's hard.
replies(1): >>45676189 #
1. typpilol ◴[22 Oct 25 22:55 UTC] No.45676189[source]▶
It will require like 20x the compute
replies(3): >>45676906 #>>45676935 #>>45676964 #
2. Mehvix ◴[23 Oct 25 00:38 UTC] No.45676906[source]▶
Why do you suppose this is a compute limited problem?
replies(1): >>45677057 #
3. ACCount37 ◴[23 Oct 25 00:42 UTC] No.45676935[source]▶
A lot of cool things are shot down by "it requires more compute, and by a lot, and we're already compute starved on any day of the week that ends in y, so, not worth it".

If we had a million times the compute? We might have brute forced our way to AGI by now.

replies(1): >>45676998 #
4. kenjackson ◴[23 Oct 25 00:48 UTC] No.45676964[source]▶
Why so much compute? Can you tie it to the problem?
replies(1): >>45679126 #
5. Jensson ◴[23 Oct 25 00:52 UTC] No.45676998[source]▶
But we don't have a million times the compute, we have the compute we have so its fair to argue that we want to prioritize other things.
6. ACCount37 ◴[23 Oct 25 01:04 UTC] No.45677057[source]▶
It's kind of a shortcut answer by now. Especially for anything that touches pretraining.

"Why aren't we doing X?", where X is a thing that sounds sensible, seems like it would help, and does indeed help, and there's even a paper here proving that it helps.

The answer is: check the paper, it says there on page 12 in a throwaway line that they used 3 times the compute for the new method than for the controls. And the gain was +4%.

A lot of promising things are resource hogs, and there are too many better things to burn the GPU-hours on.

replies(1): >>45679116 #
7. typpilol ◴[23 Oct 25 07:16 UTC] No.45679116{3}[source]▶
Thanks.

Also, saying it needs 20x compute is exactly that. It's something we could do eventually but not now

8. typpilol ◴[23 Oct 25 07:18 UTC] No.45679126[source]▶
Tokenizers are the reason LLMs are even possible to run at a decent speed on our best hardware.

Removing the tokenizer would 1/4 the context and 4x the compute and memory, assuming an avg token length of 4.

Also, you would probably need to 4x the parameters to have to learn understanding between individual characters as well as words and sentences etc.

There's been a few studies on small models, even then those only show a tiny percentage gain over tokenized models.

So essentially you would need 4x compute, 1/4 the context, and 4x the parameters to squeeze 2-4% more performance out of it.

And that fails when you use more then 1/4 context. So realistically you need to support the same context, so you r compute goes up another 4x to 16x.

That's why