←back to thread

Show HN: Qwen-2.5-32B is now the best open source OCR model

(github.com)
208 points themanmaran | 3 comments | 01 Apr 25 17:00 UTC | HN request time: 0.729s | source

Last week was big for open source LLMs. We got:

- Qwen 2.5 VL (72b and 32b)

- Gemma-3 (27b)

- DeepSeek-v3-0324

And a couple weeks ago we got the new mistral-ocr model. We updated our OCR benchmark to include the new models.

We evaluated 1,000 documents for JSON extraction accuracy. Major takeaways:

- Qwen 2.5 VL (72b and 32b) are by far the most impressive. Both landed right around 75% accuracy (equivalent to GPT-4o’s performance). Qwen 72b was only 0.4% above 32b. Within the margin of error.

- Both Qwen models passed mistral-ocr (72.2%), which is specifically trained for OCR.

- Gemma-3 (27B) only scored 42.9%. Particularly surprising given that it's architecture is based on Gemini 2.0 which still tops the accuracy chart.

The data set and benchmark runner is fully open source. You can check out the code and reproduction steps here:

- https://getomni.ai/blog/benchmarking-open-source-models-for-...

- https://github.com/getomni-ai/benchmark

- https://huggingface.co/datasets/getomni-ai/ocr-benchmark

1. pmarreck ◴[01 Apr 25 23:20 UTC] No.43552325[source]▶
Downloading the MLX version of "Qwen2.5-VL-32b-Instruct -8bit" via LM Studio right now since it's not yet available on Ollama and I can run it locally... I have an OCR side project for it to work on, want to see how performant it is on my M4... will report back
replies(1): >>43555254 #
2. hdjjhhvvhga ◴[02 Apr 25 10:19 UTC] No.43555254[source]▶
I'm very curious about the results - I've been using mistral-ocr for the last 2 weeks and it worked really well.
replies(1): >>43559088 #
3. pmarreck ◴[02 Apr 25 17:27 UTC] No.43559088[source]▶
Its errors are interesting (averaging around one per paragraph). Semantically-correct, but wrong on precision (simple example, the English word "ardour" is transcripted as "ardor", and a foreign word like "palazzo" which is intended to remain so, is translated to "palace"). I'm still messing with temp/presence/frequency/top-p/top-k/prompting to see if I can squeeze some more precision out of it, but I'm running out of time.

Not sure if it matters but I exported a PDF page as a PNG with 200dpi resolution, and used that.

It seems like it's reading the text but getting the details wrong.

I would not be comfortable using this in an official capacity without more accuracy. I could see using this for words that another OCR system is uncertain about, though, as a fallback.