Elamur Research⌕
☰
home / papers / 2026 / A Multi-Modal Recognition Framework for Ancient Books Integrating DoRA-DPO Text Recognition and YOLO Layout Analysis

A Multi-Modal Recognition Framework for Ancient Books Integrating DoRA-DPO Text Recognition and YOLO Layout Analysis

Paper · 2026 · Chaokun Zhang, Xin Wen and Tongtong Zhou

Summary

This paper presents a system entered in the closed-modality track of the shared task EvaHan 2026, which concerns multi-modal recognition of ancient Chinese documents. The task is hard because of varied scripts, rare and variant characters, degraded images and complex page layouts. The authors use two separate pipelines.

For character recognition on woodblock prints (Task A) and handwriting (Task C), they fine-tune the classical-Chinese-adapted Xunzi_Qwen2_VL_7B_Instruct with DoRA on a 4-bit base. Training follows a curriculum: woodblock data comes first, then a mixed stage that adds handwriting. They then apply DPO on cropped character tiles to fix rare-character errors. At inference, several rounds with retries and alternative prompts recover failed outputs.

For layout detection (Task B), they train a domain-pre-trained DocLayout-YOLO with Mosaic augmentation.

The system clearly beats the baselines on woodblock OCR and on layout detection. On handwriting, it improves over Xunzi but stays slightly behind Qwen2.5-VL. The code is publicly released.

Key points

  • Task A (woodblock): CER is 0.0798 with variant characters counted, versus 0.1014 for Qwen2.5_VL_7B_Instruct and 0.1786 for Xunzi. The comprehensive score is 0.9223.
  • Task C (handwriting): CER is 0.1322 (score 0.8692). This beats Xunzi (0.1497) but trails Qwen2.5-VL (0.1207 / 0.8812).
  • Task B (layout): The DocLayout-YOLOv10 system reaches mAP@[.5:.95] 0.3889, micro F1 0.6401 and IoU 0.8119. The Xunzi VLM baseline scores much lower (mAP 0.0236, IoU 0.5943).
  • DPO setup:
    • Training samples are 600×600 tiles centered on misrecognized characters.
    • A frozen reference adapter acts as a constraint.
    • Hyperparameters are β = 0.1 and a learning rate of 5×10⁻⁷.
  • Multi-round inference: The third round tries six prompts and keeps the longest output that passes filters for empty, noise and repetition errors. The authors report that this nearly eliminates empty or hallucinated outputs. The code is on GitHub at paleomindocr-evahan2026.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksEngineering · 188 worksMedia Technology · 42 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Chaokun Zhang, Xin Wen and Tongtong Zhou. 2026. A Multi-Modal Recognition Framework for Ancient Books Integrating DoRA-DPO Text Recognition and YOLO Layout Analysis. Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/58pv4t9t8hmt

@inproceedings{zhang2026multi,
  title = {{A Multi-Modal Recognition Framework for Ancient Books Integrating DoRA-DPO Text Recognition and YOLO Layout Analysis}},
  author = {Chaokun Zhang, Xin Wen and Tongtong Zhou},
  booktitle = {Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026},
  year = {2026},
  doi = {10.63317/58pv4t9t8hmt},
  url = {https://doi.org/10.63317/58pv4t9t8hmt}
}

References

8 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. model. Advances in neural information processing systems, 36:53728–53741. find
  2. Sergio Torres Aguilar. 2025. From codicology to code: A comparative study of transformer and yolo-based detectors for layout analysis in historical documents. arXiv preprint arXiv:2506.20326. — [arXiv:2506.20326](https://arxiv.org/abs/2506.20326) find
  3. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR. find
  4. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3.
  5. Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. Ocr-free document understanding transformer. In European Conference on Computer Vision, pages 498– 517. Springer.
  6. Ao Wang, Hui Chen, Lihao Liu, Kai Chen, Zijia Lin, Jungong Han, et al. 2024a. Yolov10: Real-time end-to-end object detection. Advances in neural information processing systems, 37:107984– 108011. find
  7. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
  8. Zhiyuan Zhao, Hengrui Kang, Bin Wang, and Conghui He. 2024. Doclayout-yolo: Enhancing document layout analysis through diverse synthetic data and global-to-local adaptive perception. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-multi-modal-recognition-framework-ancient/