Elamur Research⌕
☰
home / papers / 2026 / Draft, Verify, Restore: Self-Refining Historical Inscription Restoration with a Unified MLLM

Draft, Verify, Restore: Self-Refining Historical Inscription Restoration with a Unified MLLM

Paper · 2026 · Yuyi Zhang, Junle Liu, Peirong Zhang et al.

Summary

Many ancient steles carry inscriptions that fractures, erosion and oxidation have made partly unreadable. Manual restoration can take days or months for a single stele. Earlier automated approaches chain separate components, such as a language model for the text and a diffusion model for the image. These pipelines cannot correct early mistakes, and they repair images patch by patch, which weakens style consistency across the page. The authors propose UniHIR, a single unified MLLM built on BAGEL that performs the whole task end to end. It works in three steps:

  1. It first produces a visual draft marking damaged areas.
  2. Guided by this draft, it predicts the missing characters with bounding boxes. It then corrects itself at two levels: globally (source text, date, engraver) and locally (boxes and character content).
  3. It finally renders the full restored page.

To make training feasible with high-resolution images and long sequences, the authors built UHIRFactory, a memory-aware step-wise training framework. They also created HIRBench, a dataset with step-level annotations. UniHIR outperforms prior methods in both text accuracy and visual quality. This suggests that one unified model can replace multi-stage pipelines and serve as a practical aid for historians.

Key points

  • Base model: UniHIR is initialized from BAGEL, a Mixture-of-Transformer-Experts design. It combines a SigLIP2 encoder, a FLUX VAE and two Qwen2.5-based Transformers, and generates images via Rectified Flow.
  • Main result: On heavily damaged steles, OCR accuracy rises from 55.68% to 86.65% after restoration with UniHIR.
  • Ablation: Adding Draft-Guided Localization, HSR training data, and then micro- and macro-refinement at inference raises localization F1 from 67.60 to 91.42. Prediction accuracy rises from 64.24 to 96.38.
  • Comparison with AutoHDR: UniHIR has lower localization F1 than AutoHDR but higher OCR accuracy. The authors credit better semantic reconstruction and page-level consistency.
  • Limitation: A character judged damaged is fully redrawn as a standard glyph, even if some original strokes survive. Stroke-preserving repair is left for future work.
  • Availability: Code and model are released at github.com/ZZXF11/UniHIR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Yuyi Zhang, Junle Liu, Peirong Zhang, Jianliang Liu, Zhenhua Yang and Lianwen Jin. 2026. Draft, Verify, Restore: Self-Refining Historical Inscription Restoration with a Unified MLLM. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1254

@inproceedings{zhang2026draft,
  title = {{Draft, Verify, Restore: Self-Refining Historical Inscription Restoration with a Unified MLLM}},
  author = {Yuyi Zhang, Junle Liu, Peirong Zhang, Jianliang Liu, Zhenhua Yang and Lianwen Jin},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1254},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1254}
}

References

55 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yannis Assael, Thea Sommerschield, Alison Cooley, Brendan Shillingford, John Pavlopoulos, Priyanka Suresh, Bailey Herms, Justin Grayston, Benjamin Maynard, Nicholas Dietrich, and 1 others. 2025. Contextualizing ancient texts with generative neural networks. Nature, 645(8079):141–147. find
  2. Yannis Assael, Thea Sommerschield, Brendan Shillingford, Mahyar Bordbar, John Pavlopoulos, Marita Chatzipanagiotou, Ion Androutsopoulos, Jonathan Prag, and Nando de Freitas. 2022. Restoring and attributing ancient texts using deep neural networks. Nature, 603(7900):280–283. find
  3. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. Qwen3-vl technical report. arXiv prepr…
  4. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. Qwen2.5-vl technical report. Preprint, arXiv:2502.1…
  5. David Bamman and Patrick J Burns. 2020. Latin bert: A contextual language model for classical philology. arXiv preprint arXiv:2009.10053. find
  6. Black Forest Labs. 2024. Https://github.com/black-forest-labs/flux. find
  7. Jiahuan Cao, Yang Liu, Peirong Zhang, Yongxin Shi, Kai Ding, and Lianwen Jin. 2025. Tonggu-vl: Advancing visual-language understanding in chinese classical studies through parameter sensitivity-guided instruction tuning. In Proceedings of the ACM International Conference on Multimedia (MM), page 111… find
  8. Fengjiao Chen, Minhao Jing, Weitao Lu, Yan Feng, Xiaoyu Li, and Xuezhi Cao. 2025a. Unihetero: Could generation enhance understanding for vision-language-model at large data scale? arXiv preprint arXiv:2512.23512. — [arXiv:2512.23512](https://arxiv.org/abs/2512.23512) find
  9. Jiuhai Chen, Le Xue, Zhiyang Xu, Xichen Pan, Shusheng Yang, Can Qin, An Yan, Honglu Zhou, Zeyuan Chen, Lifu Huang, Tianyi Zhou, Junnan Li, Silvio Savarese, Caiming Xiong, and Ran Xu. 2025b. Blip3o-next: Next frontier of native image generation. Preprint, arXiv:2510.15857. — [arXiv:2510.15857](https://arxiv.org/abs/2510.15857) find
  10. Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. 2025. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683. — [arXiv:2505.14683](https://arxiv.org/abs/2505.14683) find
  11. Wei-Sheng Du. 1999. “restoration as old” and “restoration as new” in the repair of ancient books. Journal of Beijing Library, (04):99–102. find
  12. Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, and 1 others. 2024. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning. find
  13. Lijie Fan, Luming Tang, Siyang Qin, and et al. 2025. Unified autoregressive visual generation and understanding with continuous tokens. arXiv preprint arXiv:2503.13436. — [arXiv:2503.13436](https://arxiv.org/abs/2503.13436) find
  14. Ethan Fetaya, Yonatan Lifshitz, Elad Aaron, and Shai Gordin. 2020. Restoration of fragmentary babylonian texts using recurrent neural networks. Proceedings of the National Academy of Sciences, 117(37):22743–22751.
  15. Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. 2024. Seed-X: Multimodal Models with Unified Multi-Granularity Comprehension and Generation. arXiv preprint arXiv:2404.14396. — [arXiv:2404.14396](https://arxiv.org/abs/2404.14396) find
  16. Kaixin Han, Weitao You, Huanghuang Deng, Lingyun Sun, Jinyu Song, Zijin Hu, and Heyang Yi. 2024. Lant: finding experts for digital calligraphy character restoration. Multimedia Tools and Applications, pages 1–24. find
  17. Xin He, Longhui Wei, Jianbo Ouyang, Lingxi Xie, and Qi Tian. 2025. EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecture. arXiv preprint arXiv:2512.04810. — [arXiv:2512.04810](https://arxiv.org/abs/2512.04810) find
  18. Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Müller, Dustin Podell, Robin Rombach, Harry Saini, and 2 others. 2025. Fl… — [arXiv:2506.15742](https://arxiv.org/abs/2506.15742) find
  19. Koren Lazar, Benny Saret, Asaf Yehudai, Wayne Horowitz, Nathan Wasserman, and Gabriel Stanovsky. 2021. Filling the gaps in ancient akkadian texts: a masked language modelling approach. arXiv preprint arXiv:2109.04513.
  20. Bin Lin, Zongjian Li, Xinhua Cheng, and et al. 2025. Uniworld: High-resolution semantic encoders for unified visual understanding and generation. arXiv preprint arXiv:2506.03147. — [arXiv:2506.03147](https://arxiv.org/abs/2506.03147) find
  21. Weihong Ma, Hesuo Zhang, Lianwen Jin, Sihang Wu, Jiapeng Wang, and Yongpan Wang. 2020. Joint layout analysis, character detection and recognition for historical document digitization. In International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 31–36. IEEE. find
  22. Yiyang Ma, Xingchao Liu, Xiaokang Chen, and et al. 2025. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. In CVPR, pages 7739–7751.
  23. Xichen Pan, Satya Narayan Shukla, Aashu Singh, and et al. 2025. Transfer between modalities with meta-queries. arXiv preprint arXiv:2504.06256. — [arXiv:2504.06256](https://arxiv.org/abs/2504.06256) find
  24. Katerina Papavassileiou, Dimitrios I Kosmopoulos, and Gareth Owens. 2023. A generative model for the mycenaean linear b script and its application in infilling text from ancient tablets. ACM Journal on Computing and Cultural Heritage, 16(3):1–25. find
  25. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15… find
  26. Yang Shi, Yuhao Dong, Yue Ding, and et al. 2025a. Re-alunify: Do unified models truly benefit from unification? a comprehensive benchmark. arXiv preprint arXiv:2509.24897. — [arXiv:2509.24897](https://arxiv.org/abs/2509.24897) find
  27. Yongxin Shi, Chongyu Liu, Dezhi Peng, Cheng Jian, Jiarong Huang, and Lianwen Jin. 2023. M5HisDoc: A Large-scale Multi-style Chinese Historical Document Analysis Benchmark. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 78483–78495. find
  28. Yongxin Shi, Dezhi Peng, Yuyi Zhang, Jiahuan Cao, and Lianwen Jin. 2025b. A large-scale dataset for chinese historical document recognition and analysis. Scientific Data, 12(1):169. find
  29. Quan Sun, Qiying Yu, Yufeng Cui, and et al. 2024a. Emu: Generative pretraining in multimodality. In ICLR. find
  30. Yutao Sun, Hangbo Bao, Wenhui Wang, Zhiliang Peng, Li Dong, Shaohan Huang, Jianyong Wang, and Furu Wei. 2024b. Multimodal latent language modeling with next-token diffusion. arXiv preprint arXiv:2412.08635. — [arXiv:2412.08635](https://arxiv.org/abs/2412.08635) find
  31. Chameleon Team. 2024. Chameleon: Mixed-Modal Early-Fusion Foundation Models. arXiv preprint arXiv:2405.09818.
  32. Shengbang Tong, David Fan, Jiachen Li, and et al. 2025. MetaMorph: Multimodal Understanding and Generation via Instruction Tuning. In ICCV, pages 17001– 17012. find
  33. Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. 2025. Siglip 2: Multilingual vision-language encoders with improv…
  34. Guo-Qiang Wang. 2021. Minimal Intervention Principle for Ancient Book Conservation in China :Techniques and Application Strategy. Library Forum, 41(07):141–148. find
  35. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265.
  36. Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, and et al. 2024. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869. — [arXiv:2409.18869](https://arxiv.org/abs/2409.18869) find
  37. Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, and 20 others. 2025a. Qwen-image technical report.
  38. Chengyue Wu, Xiaokang Chen, Zhiyu Wu, and et al. 2025b. Janus: Decoupling visual encoding for unified multimodal understanding and generation. In CVPR, pages 12966–12977.
  39. Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Zhonghua Wu, Qingyi Tao, Wentao Liu, Wei Li, and Chen Change Loy. 2025c. Harmonizing visual representations for unified multimodal understanding and generation. arXiv preprint arXiv:2503.21979. — [arXiv:2503.21979](https://arxiv.org/abs/2503.21979) find
  40. Yecheng Wu, Zhuoyang Zhang, Junyu Chen, and et al. 2024. VILA-U: A Unified Foundation Model Integrating Visual Understanding and Generation. arXiv preprint arXiv:2409.04429. — [arXiv:2409.04429](https://arxiv.org/abs/2409.04429) find
  41. Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. 2025. Show-o2: Improved native unified multimodal models. arXiv preprint arXiv:2506.15564. — [arXiv:2506.15564](https://arxiv.org/abs/2506.15564) find
  42. Yi Xin, Qi Qin, Siqi Luo, Kaiwen Zhu, Juncheng Yan, Yan Tai, Jiayi Lei, Yuewen Cao, Keqi Wang, Yibin Wang, and 1 others. 2025. Lumina-dimoo: An omni diffusion large language model for multimodal generation and understanding. arXiv preprint arXiv:2510.06308. — [arXiv:2510.06308](https://arxiv.org/abs/2510.06308) find
  43. Yue Xu, Fei Yin, Da-Han Wang, Xu-Yao Zhang, Zhaoxiang Zhang, and Cheng-Lin Liu. 2019. Casia-ahcdb: A large-scale chinese ancient handwritten characters database. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 793– 798. IEEE. find
  44. Zhiyuan Yan, Kaiqing Lin, Zongjian Li, Junyan Ye, Hui Han, Zhendong Wang, Hao Liu, Bin Lin, Hao Li, Xue Xu, and 1 others. 2025. Unified multimodal model as auto-encoder. arXiv preprint arXiv:2509.09666. — [arXiv:2509.09666](https://arxiv.org/abs/2509.09666) find
  45. Zhenhua Yang, Dezhi Peng, Yongxin Shi, Yuyi Zhang, Chongyu Liu, and Lianwen Jin. 2025. Predicting the Original Appearance of Damaged Historical Documents. Proceedings of the AAAI Conference on Artificial Intelligence. find
  46. Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung-Yeung Shum. 2023. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In International Conference on Learning Representations(ICLR). find
  47. Peirong Zhang, Haowei Xu, Jiaxin Zhang, Xuhan Zheng, Guitao Xu, Yuyi Zhang, Junle Liu, Zhenhua Yang, Wei Zhou, and Lianwen Jin. 2026. OCRGen-Bench: A Comprehensive Benchmark for Evaluating OCR Generative Capabilities. find
  48. Peirong Zhang, Jiaxin Zhang, Jiahuan Cao, Hongliang Li, and Lianwen Jin. 2025a. Smaller But Better: Unifying Layout Generation with Smaller Large Language Models. International Journal of Computer Vision (IJCV), 133:3891–3917. find
  49. Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR. find
  50. Xinjie Zhang, Jintao Guo, Shanshan Zhao, Minghao Fu, Lunhao Duan, Jiakui Hu, Yong Xien Chng, Guo-Hua Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang. 2025b. Unified multimodal understanding and generation models: Advances, challenges, and opportunities. Preprint, arXiv:2505.02567. — [arXiv:2505.02567](https://arxiv.org/abs/2505.02567) find
  51. Yuyi Zhang, Yongxin Shi, Peirong Zhang, Yixin Zhao, Zhenhua Yang, and Lianwen Jin. 2025c. Mega-Han97K: A large-scale dataset for mega-category Chinese character recognition with over 97K categories. Pattern Recognition, 167:111757. find
  52. Yuyi Zhang, Peirong Zhang, Zhenhua Yang, Pengyu Yan, Yongxin Shi, Pengwei Liu, Fengjun Guo, and Lianwen Jin. 2025d. Reviving cultural heritage: A novel approach for comprehensive historical document restoration. Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  53. Yuyi Zhang, Yuanzhi Zhu, Dezhi Peng, Peirong Zhang, Zhenhua Yang, Zhibo Yang, Cong Yao, and Lianwen Jin. 2025e. Hiercode: A lightweight hierarchical codebook for zero-shot Chinese text recognition. Pattern Recognition, 158:110963. find
  54. Shipeng Zhu, Ang Chen, Na Nie, Pengfei Fang, Min-Ling Zhang, and Hui Xue. 2026. Epiagent: An agent-centric system for ancient inscription restoration. Preprint, arXiv:2604.09367. — [arXiv:2604.09367](https://arxiv.org/abs/2604.09367) find
  55. Shipeng Zhu, Hui Xue, Na Nie, Chenjie Zhu, Haiyue Liu, and Pengfei Fang. 2024. Reproducing the past: A dataset for benchmarking inscription restoration. In Proceedings of the ACM International Conference on Multimedia (MM), pages 7714–7723. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-draft-verify-restore-self-refining/