←back to thread

Karpathy on DeepSeek-OCR paper: Are pixels better inputs to LLMs than text?

(twitter.com)
233 points JnBrymn | 6 comments | 21 Oct 25 17:43 UTC | HN request time: 0.884s | source | bottom
Show context
tcdent ◴[23 Oct 25 02:16 UTC] No.45677440[source]▶
"Kill the tokenizer" is such a wild proposition but is also founded in fundamentals.

Tokenizing text is such a hack even though it works pretty well. The state-of-the-art comes out of the gate with an approximation for quantifying language that's wrong on so many levels.

It's difficult to wrap my head around pixels being a more powerful representation of information, but someone's gotta come up with something other than tokenizer.

replies(2): >>45677780 #>>45678765 #
1. dgently7 ◴[23 Oct 25 03:10 UTC] No.45677780[source]▶
I consume all text as images when I read as a vision capable person so it kinda passes the evolution does it that way test and maybe we shouldn’t be that surprised that vision is a great input method?

Actually thinking more about that I consume “text” as images and also as sounds… I kinda wonder if instead of render and ocr like this suggests we did tts and just encoded like the mp3 sample of the vocalization of the word if that would be less bytes than the rendered pixels version… probably depends on the resolution / sample rate.

replies(2): >>45678180 #>>45678258 #
2. visarga ◴[23 Oct 25 04:30 UTC] No.45678180[source]▶
Funny, I habitually read while engaging TTS on same text. I have even made a Chrome extension for web reading, it highlights text and reads it, while keeping the current position in the viewport. I find using 2 modalities at the same time improves my concentration. TTS is sped up to 1.5x to match reading speed. Maybe it is just because I want to reduce visual strain. Since I consume a lot of text every day, it can be tiring.
replies(2): >>45678346 #>>45678903 #
3. psadri ◴[23 Oct 25 04:52 UTC] No.45678258[source]▶
The pixel to sounds would pass through “reading” so there might be information loss. It is no longer just pixels.
4. lukevp ◴[23 Oct 25 05:11 UTC] No.45678346[source]▶
What’s your extension? Sounds interesting!
replies(1): >>45679286 #
5. Version467 ◴[23 Oct 25 06:44 UTC] No.45678903[source]▶
I do this too. It's great. The term I've seen used to describe this is 'Immersion Reading'. It seems to be quite a popular way for neurodivergent people to get into reading.
6. zirror ◴[23 Oct 25 07:44 UTC] No.45679286{3}[source]▶
Just FYI, Firefox reader mode does the same thing. It's a little button in the address bar.