To give an example, some models like mixtral-ocr only try to convert a document to markdown format. You have to use another LLM on top of it to get the final result. Some VLM’s directly give structured information like key fields from documents like invoices, but you have to either add business rules on top of it or use some LLM as a judge kind of system to get sense of which output needs to be manually reviewed or can be taken as correct output. No benchmark attempts to measure the actual rate of automation you can achieve.
We have tried to solve this problem with a benchmark that is only applicable for documents/usecases where you are looking for automation and its trying to measure that end to end automation level of different models or systems.
We have collected a dataset that represents documents like invoices etc which are applicable in processes where automation is needed vs are more copilot in nature where you would need to chat with document. Also have annotated these documents and published the dataset and repo so it can be extended.
Here is writeup: https://nanonets.com/automation-benchmark Dataset: https://huggingface.co/datasets/nanonets/nn-auto-bench-ds Github: https://github.com/NanoNets/nn-auto-bench
Looking for suggestions on how this benchmark can be improved further.
If you create a benchmark and your product outperforms everything else, it could mean many things. Overfitting being one of them.
Which is why publishing open benchmarks is first step where there is public scrutiny around whether then benchmark itself irrespective of the results is fair. In the end, the end user will choose the benchmark that's best fit for their usecase or mostly will create a variation of their own, do their own unbiased evaluations.