Ingesting PDFs and why Gemini 2.0 changes everything

> Why Gemini 2.0 Changes Everything

Clickbait. It doesn't change "everything". It makes ingestion for RAG much less expensive (and therefore feasible in a lot more scenarios), at the expense of ~7% reduction in accuracy. Accuracy is already rather poor even before this, however, with the top alternative clocking in at 0.9. Gemini 2.0 is 0.84, although the author seems to suggest that the failure modes are mostly around formatting rather than e.g. mis-recognition or hallucinations.

TL;DR: is this exciting? If you do RAG, yes. Does it "change everything" nope. There's still a very long way to go. Protip for model designers: accuracy is always in greater demand than performance. A slow model that solves the problem is invariably better than a fast one that fucks everything up.

In this use-case, accuracy is non-negotiable with zero room for any hallucination.

Overall it changes nothing.

And people always have a hard time understanding what a certain degree of accuracy actually means. E.g. when you hear that a speech recognition system has 95% accuracy (5% WER), it means that it gets every 19th word wrong. That's abysmally bad by human standards - errors in every other sentence. That does not mean it's useless, but you do need to understand very clearly what you're dealing with, and what those errors might do to the rest of your system.