Elamur Research⌕
☰
home / papers / 2023 / UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model

UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model

Paper · 2023 · Jiabo Ye, Anwen Hu, Haiyang Xu et al.

Summary

Understanding images that contain text, such as documents, tables, charts, webpages and street photos, usually requires either an external OCR system or costly domain-specific pretraining. Donut, for example, needed over 192 A100 days. UReader instead builds on an existing multimodal LLM, mPLUG-Owl, which already has a weak ability to recognise text without OCR. All downstream datasets from five domains are recast into one instruction-tuning format. Two auxiliary tasks in the same format, text reading and key points generation, strengthen recognition and semantic grasp.

High-resolution inputs are handled by a shape-adaptive cropping module. It picks a grid matching the image's aspect ratio and resolution, splits the image into sub-images for the frozen low-resolution encoder, adds a resized global view, and marks each crop with learnable position embeddings. Only about 1.2% of parameters are trained, via LoRA and the visual abstractor.

With no downstream finetuning, a single model reaches state-of-the-art OCR-free results on 8 of 10 benchmarks. This shows that general-purpose MLLMs can be adapted cheaply to text-rich visual understanding.

Key points

  • Base model is mPLUG-Owl, with the vision encoder and LLM frozen. Training covers LoRA adapters and the visual abstractor, about 1.2% of parameters.
  • The instruction-tuning data spans 5 domains: documents, tables, charts, natural images and webpage screenshots. Code and data are released at github.com/LukeForeverYoung/UReader.
  • Grid selection maximises the sum of two IoU scores, one resolution-related and one resolution-agnostic. In text reading with N_c = 9, the adaptive grid beats a fixed 3×3 grid on BLEU-1 (24.9 vs 21.4) and BLEU-4 (11.7 vs 9.7).
  • UReader achieves state-of-the-art OCR-free performance on 8 of 10 tasks with one jointly trained model.
  • Limitations include no multi-page document support and equal feature budgets for empty crops. The authors also note hallucination when reading long texts and weak open-ended key-point generation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China64 works · 14,113 citations🎓Renmin University of ChinaBeijing · China35 works · 14,757 citations🏢Alibaba DAMO AcademyHangzhou · China28 works · 6,115 citations🎓East China Normal UniversityShanghai · China19 works · 1,856 citations🏢Alibaba Group (Cayman Islands)George Town · Cayman Islands15 works · 3,209 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksVision & language · 269 works

Cite

Cite as Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Lin and Fei Huang. 2023. UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.187

@inproceedings{ye2023ureader,
  title = {{UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model}},
  author = {Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Lin and Fei Huang},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.187},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.187}
}

References

36 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. find
  2. Ali Furkan Biten, Rubèn Tito, Andrés Mafla, Lluís Gómez i Bigorda, Marçal Rusiñol, C. V. Jawahar, Ernest Valveny, and Dimosthenis Karatzas. 2019. Scene text visual question answering. In ICCV, pages 4290–4300. IEEE. find
  3. Lukasz Borchmann, Michal Pietruszka, Tomasz Stanislawek, Dawid Jurkiewicz, Michal Turski, Karolina Szyndler, and Filip Gralinski. 2021. DUE: end-to-end document understanding benchmark. In NeurIPS Datasets and Benchmarks. find
  4. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. Tabfact : A large-scale dataset for table-based fact verification. In International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia.
  5. Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021. Websrc: A dataset for web-based structural reading comprehension. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4173–4185. find
  6. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. CoRR, abs/2305.06500.
  7. Brian L. Davis, Bryan S. Morse, Brian L. Price, Chris Tensmeyer, Curtis Wigington, and Vlad I. Morariu. 2022. End-to-end document recognition and understanding with dessurt. In ECCV Workshops (4), volume 13804 of Lecture Notes in Computer Science, pages 280–296. Springer. find
  8. Hao Feng, Zijian Wang, Ji Tang, Jinghui Lu, Wen gang Zhou, Houqiang Li, and Can Huang. 2023. Unidoc: A universal large multimodal model for simultaneous text detection, recognition, spotting and understanding. ArXiv, abs/2308.11592. — [arXiv:2308.11592](https://arxiv.org/abs/2308.11592) find
  9. Anwen Hu, Shizhe Chen, and Qin Jin. 2021. Question-controlled text-aware image captioning. In ACM Multimedia, pages 3097–3105. ACM. find
  10. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  11. Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. 2023. Language is not all you need: Aligning perception wit…
  12. Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutlmv3: Pre-training for document AI with unified text and image masking. In ACM Multimedia, pages 4083–4091. ACM.
  13. Kushal Kafle, Brian L. Price, Scott Cohen, and Christopher Kanan. 2018. DVQA: understanding data visualizations via question answering. In CVPR, pages 5648–5656. Computer Vision Foundation / IEEE Computer Society. find
  14. Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2018. Figureqa: An annotated figure dataset for visual reasoning. In ICLR (Workshop). OpenReview.net. find
  15. Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. Ocr-free document understanding transformer. In ECCV (28), volume 13688 of Lecture Notes in Computer Science, pages 498–517. Springer.
  16. Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019. OCR-VQA: visual question answering by reading text in images. In ICDAR, pages 947–952. IEEE. find
  17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  18. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In ACL (1), pages 1470–1480. The Association for Computer Linguistics.
  19. Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Jasper Uijlings, Stefan Popov, Shahab Kamali, Mat-Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas-teo Malloci, Jordi Pont-Tuset, Andreas Veit… find
  20. Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Amanpreet Singh. 2020. Textcaps: A dataset for Fangyu Liu, Julian Eisenschlos, Urvashi Khandel-image captioning with reading comprehension. In wal, Peter Shaw, Ming-Wei Chang, and Kristina ECCV (2),… find
  21. Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA models Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. that can read. In CVPR, pages 8317–8326. find
  22. Computer Hoi. 2023. BLIP-2: bootstrapping language-image Vision Foundation / IEEE. pre-training with frozen image encoders and large language models. CoRR, abs/2301.12597. Tomasz Stanislawek, Filip Gralinski, Anna Wróblewska, Dawid Lipinski, Agnieszka Kaliska, Paulina Rosal-Haotian Liu, Chunyuan Li,…
  23. Lee. 2023a. Visual instruction tuning. CoRR, Kleister: Key information extraction datasets involv-abs/2304.08485. ing long documents with complex layouts. In ICDAR (1), volume 12821 of Lecture Notes in Computer Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Science, pages 564–579.
  24. Springer. Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. 2023b. find
  25. On S Svetlichnaya. 2020. Deepform: Understand struc-the hidden mystery of ocr in large multimodal models. tured documents at scale. arXiv preprint arXiv:2305.07895.
  26. Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. 2021. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Visualmrc: Machine reading comprehension on doc-Joty, and Enamul Hoque. 2022. Chartqa: A bench-ument images. In AAAI, pages 13878–13888. AAAI mark for question answering about charts with visual… find
  27. Zhang, and Mohit Bansal. 2023. Unifying vision, Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis text, and layout for universal document processing. find
  28. Karatzas, Ernest Valveny, and C. V. Jawahar. 2022. In Proceedings of the IEEE/CVF Conference on Com-Infographicvqa. In WACV, pages 2582–2591. IEEE. puter Vision and Pattern Recognition, pages 19254– 19264. Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawa-Hugo Touvron, Thibaut Lavril, Gautier Izac… find
  29. IEEE. Baptiste Rozière, Naman Goyal, Eric Hambro, Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, Faisal Azhar, et al. 2023. Llama: Open and effi-and Pratyush Kumar. 2020. Plotqa: Reasoning over cient foundation language models. arXiv preprint scientific plots. In WACV, pages 1516–1525. IEEE. arXi…
  30. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In CVPR, pages 4566–4575. IEEE Computer Society.
  31. Vicuna. 2023. Vicuna: An open chatbot impressing gpt-4. https://github.com/lm-sys/ FastChat. ↗
  32. Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei A. F. Florêncio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. In ACL/IJCNLP (1), pages 2579–2591. Association for Computation…
  33. Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florêncio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. 2021. TAP: text-aware pre-training for text-vqa and text-caption. In CVPR, pages 8751–8761. Computer Vision Foundation / IEEE. find
  34. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. 2023. mplug-owl: Modularization empowers large language models with multimodality. CoRR, abs/23…
  35. Liang Zhang, Anwen Hu, Jing Zhang, Shuo Hu, and Qin Jin. 2023. MPMQA: multimodal question answering on product manuals. CoRR, abs/2304.09660. — [arXiv:2304.09660](https://arxiv.org/abs/2304.09660) find
  36. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ye-2023-ureader-universal-ocr-free-visually/