Elamur Research⌕
☰
home / papers / 2026 / OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets

OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets

Paper · 2026 · Jiyuan Shen, Yuan Peiyue, Atin Ghosh et al.

Summary

Industrial document extraction systems usually run OCR first and then pass the recognized text to a separate extraction model. These pipelines are complex and carry OCR mistakes forward into extraction. The authors, from SAP and Stanford, ask whether modern multimodal LLMs can skip OCR entirely.

They benchmark off-the-shelf MLLMs on two internal, multilingual business-document datasets: C1 (supply chain) and C2 (finance). The extraction is zero-shot, with a schema-based prompt that returns JSON containing header fields and line items. Each model is tested with three inputs: page image only, in-house OCR text only, and both together. Scoring uses key–value F1.

The authors also build an automated hierarchical error analysis framework with three stages:

  • a rule-based error handler,
  • LLM/MLLM reasoning over the errors,
  • clustering of the resulting explanations.

The strongest models perform about as well, or better, on images alone as on OCR-enhanced input. Image-only input also produces the fewest errors. Using the error diagnosis to refine the schema, format and prompt raises Gemini 1.5 Pro further. This suggests that simpler MLLM-only pipelines are a realistic option for enterprise document AI.

Key points

  • Models evaluated:
    • Llama 4 Scout and Llama 4 Maverick
    • Pixtral Large (2411)
    • Nova Pro
    • GPT-4o and GPT-4o mini
    • Claude 3 Opus and Claude 3.5 Sonnet
    • Gemini 1.5 Pro, Gemini 2.0 Pro and Gemini 2.5 Flash
  • Best result: Gemini 1.5 Pro with image-only input scored a mean F1 of 76.8 (87.3 on C1, 66.4 on C2). This beat its OCR-only (≈74.1) and image+OCR (≈75.6) results. A refined prompt then lifted the image-only mean to 78.9.
  • Error types: The framework groups errors into three classes:
    • text misinterpretation;
    • image-to-text extraction issues, which are higher without OCR;
    • OCR/schema ambiguity, which is lower with image-only input.
  • Error clustering: Explanations are grouped using BERT-based embedding clustering with cosine similarity.
  • Cost and speed (closed-source models):
    • Gemini 2.5 Flash was fastest, at about 1.4 s per page.
    • Gemini 1.5 Pro was cheapest, at about $0.001 per page.
    • Claude 3 Opus was the slowest and most expensive, at about 7.0 s and $0.05 per page.
  • Limitations: Few-shot learning, CoT/self-reflection and reasoning models (O1, DeepSeek R1) were not systematically explored. The datasets are internal and not released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Information extraction · 167 works

Cite

Cite as Jiyuan Shen, Yuan Peiyue, Atin Ghosh, Yifan Mai and Daniel Dahlmeier. 2026. OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-industry.28

@inproceedings{shen2026ocr,
  title = {{OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets}},
  author = {Jiyuan Shen, Yuan Peiyue, Atin Ghosh, Yifan Mai and Daniel Dahlmeier},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.28},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.28}
}

References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  2. Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, et al. 2024. Automatic root cause analysis via large language models for cloud incidents. In Proceedings of the Nineteenth European Conference on Computer Systems, pages 674–688. find
  3. Gartner. Intelligent document processing solutions reviews and ratings. https://www.gartner.com/reviews/market/ intelligent-document-processing-solutions. ↗
  4. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  5. Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. LayoutLMv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM international conference on multimedia, pages 4083–4091.
  6. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  7. Amazon Artificial General Intelligence. 2024. The amazon nova family of models: Technical report and model card. find
  8. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
  9. Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. 2024. TextMonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473. — [arXiv:2403.04473](https://arxiv.org/abs/2403.04473) find
  10. MistralAI. Mistral ocr technique report. https://mistral.ai/news/mistral-ocr. ↗
  11. Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, et al. 2025. Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations. In Proceedings of the Computer Vision and Pattern Recognition Conference, pag… find
  12. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  13. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.
  14. Zilong Wang, Yichao Zhou, Wei Wei, Chen-Yu Lee, and Sandeep Tata. 2023. VRDU: A benchmark for visually-rich document understanding. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5184–5193. find
  15. Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. 2023. mPLUG-DocOwl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499. — [arXiv:2307.02499](https://arxiv.org/abs/2307.02499) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2026-ocr-or-not-rethinking-document/