Elamur Research⌕
☰
home / papers / 2026 / Benchmarking Vision-Language Models on Chinese Ancient Documents: From OCR to Knowledge Reasoning

Benchmarking Vision-Language Models on Chinese Ancient Documents: From OCR to Knowledge Reasoning

Paper · 2026 · Haiyang Yu, Yuchuan Wu, Fan Shi et al.

Summary

Large-scale scanning of historical Chinese books has produced many page images, but scanning alone does not make their content searchable or understandable. Existing document benchmarks such as DocVQA, OCRBench and CN-DocVQA mostly target modern English or simplified-Chinese print, so they cannot show how well vision-language models (VLMs) handle classical texts. The authors introduce AncientDoc, which they present as the first benchmark of its kind. It tests VLMs on five tasks that range from raw transcription to cultural knowledge, using page images drawn from more than 100 historical books.

Mainstream VLMs were scored with standard metrics, including BERTScore. GPT-4o served as an additional judge because its ratings were found to track human judgments most closely. Page-level OCR proved the hardest step: vertical layouts, interlinear notes and obsolete glyphs still defeat current models. Yet models perform noticeably better on the QA tasks, which suggests they can draw on partial evidence and prior knowledge. The benchmark gives cultural-heritage NLP a common way to diagnose where VLMs fail.

Key points

  • Coverage: AncientDoc spans five tasks, 14 document types, more than 100 books and roughly 3,000 pages.
    • The tasks are page-level OCR, rendering the text into modern vernacular Chinese, and three question-answering tasks that respectively probe reasoning, background knowledge and linguistic variants.
  • Translation is harsh under GPT-4o scoring: the best model reached only 4.72 out of 10.
    • GPT-4o gave its own translations just 0.92, which suggests it does not favor its own outputs.
    • BERTScore and GPT-4o ratings were positively correlated.
  • Scale matters little for linguistic variant QA: 2B–4B models matched or beat 7B–8B models on BS-F1.
    • The InternVL2.5 series led at every size, which points to training-data diversity rather than parameter count.
  • Different tasks fail for different reasons:
    • Translation suffers from polysemy and missing punctuation.
    • Reasoning QA depends on integrating several sentences.
    • Knowledge QA reveals gaps in historical and cultural knowledge.
  • Stated limitations:
    • Ming–Qing documents and regular script are overrepresented.
    • Evaluation is limited to single pages, with no cross-page reasoning.
    • Judgments rely on automatic and LLM-based metrics rather than expert human review.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations🏢ByteDanceBeijing · China41 works · 4,180 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksVision & language · 657 works

Cite

Cite as Haiyang Yu, Yuchuan Wu, Fan Shi, Jinghui Lu, Ke Niu, Xiaodong Ge, Minghan Zhuo, Jingqun Tang and Bin Li. 2026. Benchmarking Vision-Language Models on Chinese Ancient Documents: From OCR to Knowledge Reasoning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1438

@inproceedings{yu2026benchmarking,
  title = {{Benchmarking Vision-Language Models on Chinese Ancient Documents: From OCR to Knowledge Reasoning}},
  author = {Haiyang Yu, Yuchuan Wu, Fan Shi, Jinghui Lu, Ke Niu, Xiaodong Ge, Minghan Zhuo, Jingqun Tang and Bin Li},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1438},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1438}
}

References

40 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alireza Akoushideh, Atefeh Ranjkesh Rashtehroudi, and Asadollah Shahbahrami. 2025. Persian/arabic scene text recognition with convolutional recurrent neural network. IET Smart Cities, 7(1):e70001. find
  2. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966.
  3. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. Qwen3-vl technical report. Preprint, a…
  4. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025b. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  5. Łukasz Borchmann, Michał Pietruszka, Tomasz Stanislawek, Dawid Jurkiewicz, Michał Turski, Karolina Szyndler, and Filip Graliński. 2021. Due: End-to-end document understanding benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). find
  6. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, and 1 others. 2024. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271.
  7. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  8. Ran Cui, Aichun Zhu, and Zichen Ding. 2025. Enhanced chinese scene text recognition model base on cross-domain feature fusion. Signal, Image and Video Processing, 19(6):499. find
  9. Yihao Ding, Siwen Luo, Hyunsuk Chung, and Soyeon Caren Han. 2023. Vqa: A new dataset for real-world vqa on pdf documents. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 585–601. Springer. find
  10. Yihao Ding, Kaixuan Ren, Jiabin Huang, Siwen Luo, and Soyeon Caren Han. 2024. Mvqa: A dataset for multimodal information retrieval in pdf-based visual question answering. arXiv preprint arXiv:2404.12720. — [arXiv:2404.12720](https://arxiv.org/abs/2404.12720) find
  11. Ling Fu, Zhebin Kuang, Jiajun Song, Mingxin Huang, Biao Yang, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, and 1 others. 2024. Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning. arXiv preprint arXiv:2501.00321. — [arXiv:2501.00321](https://arxiv.org/abs/2501.00321) find
  12. Pan He, Weilin Huang, Yu Qiao, Chen Loy, and Xiaoou Tang. 2016. Reading scene text in deep convolutional sequences. In Proceedings of the AAAI conference on artificial intelligence, volume 30. find
  13. Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutlmv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM international conference on multimedia, pages 4083–4091.
  14. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  15. R Reeve Ingle, Yasuhisa Fujii, Thomas Deselaers, Jonathan Baccash, and Ashok C Popat. 2019. A scalable handwritten text recognition system. In 2019 International conference on document analysis and recognition (ICDAR), pages 17–24. IEEE. find
  16. Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pages 1–6. IEEE. find
  17. Lei Kang, Pau Riba, Marçal Rusiñol, Alicia Fornés, and Mauricio Villegas. 2022. Pay attention to what you read: non-recurrent handwritten text-line recognition. Pattern Recognition, 129:108766. find
  18. Lei Kang, Rubèn Tito, Ernest Valveny, and Dimosthenis Karatzas. 2024. Multi-page document visual question answering using self-attention scoring mechanism. In International Conference on Document Analysis and Recognition, pages 219–232. Springer. find
  19. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306.
  20. Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. 2024b. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):220102.
  21. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, and 1 others. 2024. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. Advances in Neural Information Processing Systems, 37:95963–96010. find
  22. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209.
  23. Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE. find
  24. Maja Popović. 2017. chrf++: words helping character n-grams. In Proceedings of the second conference on machine translation, pages 612–618.
  25. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2024. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  26. Ryota Tanaka, Taichi Iki, Kyosuke Nishida, Kuniko Saito, and Jun Suzuki. 2024. Instructdoc: A dataset for zero-shot generalization of visual document understanding with instructions. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 19071–19079. find
  27. ByteDance Seed Team. 2025. Seed1.5-vl technical report. arXiv preprint arXiv:2505.07062.
  28. Fei Yin, Yi-Chao Wu, Xu-Yao Zhang, and Cheng-Lin Liu. 2017. Scene text recognition with sliding convolutional character models. arXiv preprint arXiv:1709.01727. — [arXiv:1709.01727](https://arxiv.org/abs/1709.01727) find
  29. Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. arXiv preprint arXiv:1810.03581.
  30. Jinxu Zhang, Yongqi Yu, and Yu Zhang. 2024. Cream: coarse-to-fine retrieval and multi-modal efficient tuning for document vqa. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 925–934. find
  31. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. find
  32. Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. 2017. East: an efficient and accurate scene text detector. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 5551–5560. find
  33. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, and 1 others. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479.
  34. Zhi Tian, Weilin Huang, Tong He, Pan He, and Yu Qiao. 2016. Detecting text in natural image with connectionist text proposal network. In European conference on computer vision, pages 56–72. Springer. find
  35. A Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023. Document-level machine translation with large language models. arXiv preprint arXiv:2304.02210.
  36. The 14 categories of Chinese ancient documents included in AncientDoc cover a wide range of genres, reflecting the diversity of knowledge and literary forms in traditional Chinese culture. Their specific explanations are as follows: Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai… — DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
  37. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, and 1 others. 2025. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215. — Qwen2.5-Omni Technical Report
  38. Zhongguo Xu and Yang Xiang. 2025. Improving chinese word segmentation with character–lexicon class attention. Neural Computing and Applications, 37(5):3857–3867. find
  39. Hailin Yang, Lianwen Jin, Weiguo Huang, Zhaoyang Yang, Songxuan Lai, and Jifeng Sun. 2018. Dense and tight detection of chinese characters in historical documents: Datasets and a recognition guided detector. IEEE Access, 6:30174–30183. find
  40. Explanations of 14 Document Categories in AncientDoc • Biography: These texts focus on recording the life stories, achievements, and moral virtues of historical figures, typically presenting chronological narratives that serve as moral examples for readers. • Confucianism: Works centered on Confucia… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2026-benchmarking-vision-language-models-chinese/