Elamur Research⌕
☰
home / papers / 2026 / Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation

Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation

Paper · 2026 · Jianing Zhang, Runan Li, Honglin Pang et al.

Summary

Most AI work on Oracle Bone Script (OBS) treats decipherment as closed-set image classification. That approach cannot handle rare or unseen glyphs, and general Vision-Language Models (VLMs) tend to hallucinate in this low-resource domain. The authors instead treat interpretation as reasoning over a character's parts. OBS characters are built from a limited set of reused pictographic components, and each component carries meaning that transfers across characters.

The authors propose an Agentic RAG pipeline with four steps:

  1. A VLM grounds the visual components.
  2. An LLM agent queries a knowledge graph of components, characters and explanations.
  3. The system infers how the components relate.
  4. It generates a full interpretation.

To support this pipeline, they built OB-Radix, an expert-annotated component-level dataset. They evaluate on three benchmarks of increasing difficulty: component retrieval, relationship inference and interpretation generation. Across all three, the framework produces more precise and detailed readings than plain VLMs. Gains also carry over to English output. The work matters because two-thirds of known OBS characters remain undeciphered, and interpretations that cite traceable evidence can assist paleographers.

Key points

  • OB-Radix dataset: 1,022 character images (934 unique characters) and 1,853 component images covering 478 distinct components, each with an expert-verified explanation. Three archaeology PhD students produced the annotations using LabelMe masks, in about 70 man-hours.
  • Annotation principles: components were separated by semantic role rather than visual size. Semantic integrity took priority over geometric completeness, and labels came from a controlled vocabulary.
  • Pipeline design: a DINOv2 Base encoder matches components to prototypes. The agent then retrieves knowledge-graph entries through graph queries (Graph RAG). The approach was tested with GPT, Claude, GLM and Qwen backbones.
  • English interpretation task (LLM-Judge scores): scores rose sharply over the baselines, for example GPT from 0.098 to 0.797 and GLM from 0.051 to 0.489. Absolute scores are lower than in Chinese because the knowledge graph is Chinese-centric.
  • Limitations: component detection is imperfect, and errors in the knowledge graph can propagate into interpretations. Phono-semantic compounds and variant-character recognition remain hard. Code is available at github.com/6zjn6/OBS-expert.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of TokyoTokyo · Japan90 works · 9,156 citations🎓Jilin UniversityChangchun · China25 works · 5,252 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Jianing Zhang, Runan Li, Honglin Pang, Ding Xia, Zhou Zhu, Qian Zhang, Chuntao Li and Xi Yang. 2026. Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1626

@inproceedings{zhang2026specializing,
  title = {{Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation}},
  author = {Jianing Zhang, Runan Li, Honglin Pang, Ding Xia, Zhou Zhu, Qian Zhang, Chuntao Li and Xi Yang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1626},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1626}
}

References

50 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. System card addendum: Claude opus 4.1. Technical report, Anthropic. find
  2. Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. 2024. The revolution of multimodal large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2024, pages…
  3. Chia-Yuan Chang, Zhimeng Jiang, Vineeth Rakesh, Menghai Pan, Chin-Chia Michael Yeh, Guanchu Wang, Mingzhi Hu, Zhichao Xu, Yan Zheng, Ma-hashweta Das, and 1 others. 2024. Main-rag: Multi-agent filtering retrieval-augmented generation. arXiv preprint arXiv:2501.00332.
  4. Zijian Chen, tingzhu chen, Wenjun Zhang, and Guangtao Zhai. 2025. OBI-bench: Can LMMs aid in study of ancient script on oracle bones? In The Thirteenth International Conference on Learning Representations. find
  5. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In Intern…
  6. Xuanming Fu, Zhengfeng Yang, Zhenbing Zeng, Yidan Zhang, and Qianting Zhou. 2022. Improvement of oracle bone inscription recognition accuracy: A deep learning perspective. ISPRS International Journal of Geo-Information, 11(1):45. find
  7. Ji Gan, Yuyan Chen, Bo Hu, Jiaxu Leng, Weiqiang Wang, and Xinbo Gao. 2023. Characters as graphs: Interpretable handwritten chinese character recognition via pyramid graph transformer. Pattern Recognition, 137:109317. find
  8. Shuangping Huang, Haobin Wang, Yongge Liu, Xiaosong Shi, and Lianwen Jin. 2019. Obc306: A large-scale oracle bone character recognition dataset. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 681–688. IEEE. find
  9. Hanqi Jiang, Yi Pan, Junhao Chen, Zhengliang Liu, Yifan Zhou, Peng Shu, Yiwei Li, Huaqin Zhao, Stephen Mihm, Lewis C Howe, and 1 others. 2024. Or-aclesage: Towards unified visual-linguistic understanding of oracle bone scripts through cross-modal knowledge fusion. arXiv preprint arXiv:2411.17837. — [arXiv:2411.17837](https://arxiv.org/abs/2411.17837) find
  10. Runhua Jiang, Yongge Liu, Boyuan Zhang, Xu Chen, Deng Li, and Yahong Han. 2023. Oraclepoints: A hybrid neural representation for oracle character. In Proceedings of the 31st ACM international conference on multimedia, pages 7901–7911. find
  11. Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. 2024. Ragcache: Efficient knowledge caching for retrieval-augmented generation. arXiv preprint arXiv:2404.12457. find
  12. Mingyu Jin, Weidi Luo, Sitao Cheng, Xinyi Wang, Wenyue Hua, Ruixiang Tang, William Yang Wang, and Yongfeng Zhang. 2025. Disentangling memory and reasoning ability in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape…
  13. Haisu Guan, Jinpeng Wan, Yuliang Liu, Pengjie Wang, Kaile Zhang, Zhebin Kuang, Xinyu Wang, Xiang Bai, and Lianwen Jin. 2024a. An open dataset for the evolution of oracle bone characters: Evobc. Preprint, arXiv:2401.12467. — [arXiv:2401.12467](https://arxiv.org/abs/2401.12467) find
  14. Bang Li, Qianwen Dai, Feng Gao, Weiye Zhu, Qiang Li, and Yongge Liu. 2020. Hwobc-a handwriting oracle bone character recognition database. In Journal of Physics: Conference Series, volume 1651, page 012050. IOP Publishing. find
  15. Haisu Guan, Huanxin Yang, Xinyu Wang, Shengwei Han, Yongge Liu, Lianwen Jin, Xiang Bai, and Yuliang Liu. 2024b. Deciphering oracle bone language with diffusion models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15554– 155… find
  16. Bang Li, Donghao Luo, Yujie Liang, Jing Yang, Zengmao Ding, Xu Peng, Boyuan Jiang, Shengwei Han, Dan Sui, Peichao Qin, Pian Wu, Chaoyang Wang, Yun Qi, Taisong Jin, Chengjie Wang, Xiaoming Huang, Zhan Shu, Rongrong Ji, Yongge Liu, and Yunsheng Wu. 2024. Oracle bone inscriptions multimodal dataset. Pr… — [arXiv:2407.03900](https://arxiv.org/abs/2407.03900) find
  17. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Wu Z. F. Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 174 others. 2025. Deepseek-r1 incentivizes reasoning in llms through reinf… find
  18. Caoshuo Li, Zengmao Ding, Xiaobin Hu, Bang Li, Donghao Luo, AndyPian Wu, Chaoyang Wang, Chengjie Wang, Taisong Jin, Seven Shu, and 1 others. 2025a. Oraclefusion: Assisting the decipherment of oracle bone script with structurally constrained semantic typography. In Proceedings of the IEEE/CVF Interna… find
  19. Wenhui Han, Xinlin Ren, Hangyu Lin, Yanwei Fu, and Xiangyang Xue. 2020. Self-supervised learning of orc-bert augmentator for recognizing few-shot oracle characters. In Proceedings of the Asian Conference on Computer Vision. find
  20. Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. 2025b. Preference leakage: A contamination problem in llm-as-a-judge. arXiv preprint arXiv:2502.01534. — [arXiv:2502.01534](https://arxiv.org/abs/2502.01534) find
  21. Zhikai Hu, Yiu-ming Cheung, Yonggang Zhang, Peiying Zhang, and Pui-ling Tang. 2024. Component-level oracle bone inscription retrieval. In Proceedings of the 2024 International Conference on Multimedia Retrieval, pages 647–656. find
  22. Zhiheng Li, Zongyang Ma, Yuntong Pan, Ziqi Zhang, Xiaolei Lv, Bo Li, Jun Gao, Jianing Zhang, Chunfeng Yuan, Bing Li, and Weiming Hu. 2026. Making mllms blind: Adversarial smuggling attacks in mllm content moderation. Preprint, arXiv:2604.06950. find
  23. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  24. Demiao Lin. 2024. Revolutionizing retrieval-augmented generation with enhanced pdf structure recognition. arXiv preprint arXiv:2401.12599. — [arXiv:2401.12599](https://arxiv.org/abs/2401.12599) find
  25. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916.
  26. Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196.
  27. Thang Nguyen, Peter Chin, and Yu-Wing Tai. 2025. Ma-rag: Multi-agent retrieval-augmented generation via collaborative chain-of-thought reasoning. arXiv preprint arXiv:2505.20096.
  28. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, and 7 o… — DINOv2: Learning Robust Visual Features without Supervision
  29. Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. 2024. Graph retrieval-augmented generation: A survey. arXiv preprint arXiv:2408.08921.
  30. Runqi Qiao, Lan Yang, Kaiyue Pang, and Honggang Zhang. 2024. Making visual sense of oracle bones for you and me. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12656–12665. find
  31. Qwen Team. 2025. Qwen3-VL: A more powerful large-scale vision-language model. https://qwen.ai/blog?id= 99f0335c4ad9ff6153e517418d48535ab6d8afef& from=research.latest-advancements-list. Accessed: 2025-05-28. ↗
  32. Bryan C Russell, Antonio Torralba, Kevin P Murphy, and William T Freeman. 2008. Labelme: a database and web-based tool for image annotation. International journal of computer vision, 77(1):157–173. find
  33. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539–68551.
  34. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, and et al. Ananthram, Akhila. 2025a. Gpt-5 system card. Technical report, OpenAI. find
  35. Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. 2025b. Agentic retrieval-augmented generation: A survey on agentic rag. arXiv preprint arXiv:2501.09136. — [arXiv:2501.09136](https://arxiv.org/abs/2501.09136) find
  36. Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems 30, pages 4080–4090. Curran Associates, Inc. find
  37. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  38. V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, and 69 others. 2025. Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reas…
  39. Pengjie Wang, Kaile Zhang, Xinyu Wang, Shengwei Han, Yongge Liu, Lianwen Jin, Xiang Bai, and Yuliang Liu. 2024a. Puzzle pieces picker: Deciphering ancient chinese characters with radical reconstruction. In International Conference on Document Analysis and Recognition, pages 169–187. Springer. find
  40. Pengjie Wang, Kaile Zhang, Xinyu Wang, Shengwei Han, Yongge Liu, Jinpeng Wan, Haisu Guan, Zhebin Kuang, Lianwen Jin, Xiang Bai, and 1 others. 2024b. An open dataset for oracle bone character recognition and decipherment. Scientific Data, 11(1):976. find
  41. Junde Wu, Jiayuan Zhu, Yuyuan Liu, Min Xu, and Yueming Jin. 2025. Agentic reasoning: A streamlined framework for enhancing LLM reasoning with agentic tools. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28489– 28503, Vienna,…
  42. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 3(4).
  43. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  44. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
  45. Cheng Ye. 2024. Exploring a learning-to-rank approach to enhance the retrieval augmented generation (rag)-based electronic medical records search engines. Informatics and Health, 1(2):93–99. find
  46. Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-language models for vision tasks: A survey. IEEE transactions on pattern analysis and machine intelligence, 46(8):5625–5644. find
  47. Ruixiang Zhang, Yu Wang, Weiyang Yang, Jun Wen, Weizhi Liu, Shipeng Zhi, Guangzhou Li, Nan Chai, Jiaqi Huang, Yongyao Xie, and 1 others. 2025. Plant-gpt: An arabidopsis-based intelligent agent that answers questions about plant functional genomics. Advanced Science, page e03926. find
  48. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  49. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International…
  50. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-specializing-large-models-oracle-bone/