Elamur Research⌕
☰
home / papers / 2026 / LVLM Optimization for Ancient Chinese Book Image Analysis with Task-specific Augmentation and Instruction Tuning

LVLM Optimization for Ancient Chinese Book Image Analysis with Task-specific Augmentation and Instruction Tuning

Paper · 2026 · Tian Xia, Yulong Liu, Yilin Wang et al.

Summary

The paper addresses OCR and layout analysis for digitized ancient Chinese books, where variant characters, damaged strokes, mixed printed and handwritten scripts, and crowded page structures (text blocks, margins, seals, illustrations) confuse conventional OCR. Working within the EvaHan 2026 shared task, the authors adapt Qwen2.5-VL-7B-Instruct with LoRA through the ms-swift framework. Qwen2.5-VL-7B-Instruct was chosen over Xunzi_Qwen2_VL_7B_Instruct after preliminary comparisons. The system covers three datasets: printed text (A), mixed layouts (B) and handwriting (C). Each task gets its own augmentation and its own prompt template, so that transcription and bounding-box prediction are handled separately.

For text recognition, the closed track uses geometric augmentation and the open track uses stitched print and handwriting composites. For layout, the closed track places pages on random backgrounds, while the open track uses a fine-tuned DocLayout-YOLO. The tuned systems beat all official baselines, and zero-shot LVLMs performed poorly. Task-aware adaptation therefore makes general LVLMs practical for heritage digitization. Layout localization remains weak for the LVLM, and no ablations are reported because the test labels are withheld.

Key points

  • Augmentation:
    • Closed-track OCR: flips, affine transforms and random crops produce 4–5 variants per image, giving more than 20,000 samples.
    • Open-track OCR: 10,000 samples are built by stacking printed and handwritten images vertically, in both orders.
    • Closed-track layout: 10,000 samples are made by placing pages on a light 924×1232 canvas, either at the top-left corner or at a random position.
  • Fine-tuning setup: LoRA with rank 8 and alpha 32 is applied to all linear layers. The ViT encoder is frozen because unfreezing it brought no gains.
  • OCR results (CER, variant-aware):
    • Task A (printed): 0.0372, against 0.0618 for plain LoRA and 0.1014 for zero-shot Qwen.
    • Task C (handwritten): 0.0823, against 0.0920 for plain LoRA.
  • Layout (Task B, mAP@[.5:.95]):
    • Background-embedding LVLM: 0.2933.
    • DocLayout-YOLO (open track): 0.3828, with Micro F1 of 0.6673.
    • Zero-shot Qwen: 0.
  • Availability: the code and prompt templates are released at github.com/iamxiatian/evahan2026.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Renmin University of ChinaBeijing · China139 works · 17,752 citations🎓Washington University in St. LouisSt Louis · United States15 works · 1,129 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksEngineering · 188 worksMedia Technology · 42 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Tian Xia, Yulong Liu, Yilin Wang, Yumeng Yang, Dongheng Cai, Yuyang Tan and Menghui Yang. 2026. LVLM Optimization for Ancient Chinese Book Image Analysis with Task-specific Augmentation and Instruction Tuning. Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3w5rwjqr49n7

@inproceedings{xia2026lvlm,
  title = {{LVLM Optimization for Ancient Chinese Book Image Analysis with Task-specific Augmentation and Instruction Tuning}},
  author = {Tian Xia, Yulong Liu, Yilin Wang, Yumeng Yang, Dongheng Cai, Yuyang Tan and Menghui Yang},
  booktitle = {Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026},
  year = {2026},
  doi = {10.63317/3w5rwjqr49n7},
  url = {https://doi.org/10.63317/3w5rwjqr49n7}
}

References

10 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Keqin Chen, Xuejing Liu, et al. 2025. Qwen2.5-VL technical report.
  2. Edward J. Hu, Yelong Shen, Phillip Wallis, et al. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  3. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. 2023. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.
  4. Minghao Li, Tengchao Lv, Jingye Chen, et al. 2023. TrOCR: Transformer-based optical character recognition with pre-trained models. In Proceedings of the 37th AAAI Conference on Artificial Intelligence, pages 13094–13102. find
  5. Chuwei Luo, Yufan Shen, Zhaoqing Zhu, et al. 2024. LayoutLLM: layout instruction tuning with large language models for document understanding. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15630– 15640. find
  6. Baoguang Shi, Xiang Bai, and Cong Yao. 2016. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(11):2298–2304. find
  7. Yongxin Shi, Dezhi Peng, Yuyi Zhang, et al. 2025. A large-scale dataset for chinese historical document recognition and analysis. Scientific Data, 12(1):169. find
  8. Dongbo Wang and Dongmei Zhu. 2025. Benchmarking the ancient books capability of multimodal large language models. Npj Heritage Science, 13(1):339. find
  9. Yuze Zhao, Jintao Huang, Jinghan Hu, et al. 2025. SWIFT: A scalable lightweight infrastructure for fine-tuning. find
  10. Zhiyuan Zhao, Hengrui Kang, Bin Wang, et al. 2024. DocLayout-YOLO: Enhancing document layout analysis through diverse synthetic data and global-to-local adaptive perception. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xia-2026-lvlm-optimization-ancient-chinese-book/