Elamur Research⌕
☰
home / papers / 2026 / ACSE: An Ancient Character Semantic-Aware Embedding for Large Language Models

ACSE: An Ancient Character Semantic-Aware Embedding for Large Language Models

Paper · 2026 · Zhihan Zhou, Daqian Shi, Lida Shi et al.

Summary

Large language models handle transmitted classical Chinese reasonably well, yet they falter on excavated writings from the pre-Qin era, including the Bamboo Book of Chu, Bronze Inscriptions, and Oracle Bone Inscriptions. These scripts are poorly digitized, have very little training data, and pack meaning into glyph components whose senses changed over time. The authors propose ACSE, an embedding module for Qwen2.5. It relies on the BBPE tokenizer, which represents ancient characters through byte indices already shared with modern Chinese, so no new tokens are needed. For subsequences matching an ancient-character vocabulary, ACSE builds a glyph embedding and a lexical (sense) embedding, fuses them, and projects the result into the modern Chinese semantic space alongside the original embedding. Training is lightweight and runs in two stages: contrastive learning on the glyph and lexical embeddings, then instruction fine-tuning of the fusion module. On tasks built from excavated documents, ACSE outperforms the original embedding and LoRA-style fine-tuning. This shows how modern LLM knowledge can be reused for extremely low-resource historical scripts.

Key points

  • Token overlap analysis: A preliminary study with the Qwen2.5 tokenizer compared 2,909 ancient characters with 8,827 modern CJK characters (U+4E00–U+9FFF). It found that 2,696 ancient characters share byte-level indices with modern ones.
  • Evaluation tasks (5-shot ACSE scores):
    • Radical: 70.01
    • Radical Meaning: 72.61
    • ExcDoc Word: 81.91
    • Phonetic Loan Character: 77.94
  • Ablation results: Removing either the glyph or the lexical embedding pulls scores back to roughly baseline levels. For example, Radical Meaning falls to about 50, which is a 22.71-point drop when the lexical embedding is removed.
  • Conventional fine-tuning fails: Standard fine-tuning that adds new token indices pushed accuracy to about 25% on the four tasks, which is near chance. The authors attribute this to damage to the model's original representations.
  • Limitations: The method is text-only and was tested only on Qwen2.5. The authors plan to try other BBPE-based models and multimodal extensions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University College LondonLondon · United Kingdom107 works · 22,923 citations🎓Queen Mary University of LondonLondon · United Kingdom57 works · 3,600 citations🎓Jilin UniversityChangchun · China30 works · 5,252 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksLiterature and Literary Theory · 541 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Zhihan Zhou, Daqian Shi, Lida Shi, Rui Song, Peiqiang Qiu, Xiaolei Diao and Hao Xu. 2026. ACSE: An Ancient Character Semantic-Aware Embedding for Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.437

@inproceedings{zhou2026acse,
  title = {{ACSE: An Ancient Character Semantic-Aware Embedding for Large Language Models}},
  author = {Zhihan Zhou, Daqian Shi, Lida Shi, Rui Song, Peiqiang Qiu, Xiaolei Diao and Hao Xu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.437},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.437}
}

References

53 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yannis Assael, Thea Sommerschield, Brendan Shillingford, Mahyar Bordbar, John Pavlopoulos, Marita Chatzipanagiotou, Ion Androutsopoulos, Jonathan Prag, and Nando De Freitas. 2022. Restoring and attributing ancient texts using deep neural networks. Nature, 603(7900):280–283. find
  2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, and Jianxin Ma. 2023. Qwen technical report. Preprint, arXiv:2309.16609.
  3. Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 1–9. Association for Computational Linguistics.
  4. William G Boltz. 1986. Early chinese writing. World Archaeology, 17(3):420–436. find
  5. Jiahuan Cao, Dezhi Peng, Yongxin Shi, Zongyuan Jiang, and Lianwen Jin. 2023. Translating ancient chinese to modern chinese at scale: A large language model-based approach. In Proceedings of ALT2023: Ancient Language Translation Workshop, pages 61–69. find
  6. Jiahuan Cao, Dezhi Peng, Peirong Zhang, Yongxin Shi, Yang Liu, Kai Ding, and Lianwen Jin. 2024. Tonggu: Mastering classical chinese understanding with knowledge-grounded large language models. Preprint, arXiv:2407.03937. find
  7. Ernie Chang, Yow-Ting Shiue, Hui-Syuan Yeh, and Vera Demberg. 2021. Time-aware ancient chinese text translation and inference. arXiv preprint arXiv:2107.03179. find
  8. Liu Chang, Wang Dongbo, Zhao Zhixiao, Hu Die, Wu Mengcheng, Lin Litao, Shen Si, Li Bin, Liu Jiangfeng, Zhang Hai, and 1 others. 2023. Sikugpt: A generative pre-trained model for intelligent information processing of ancient texts from the perspective of digital humanities. arXiv preprint arXiv:2304.… — [arXiv:2304.07778](https://arxiv.org/abs/2304.07778) find
  9. Yang Chi, Fausto Giunchiglia, Daqian Shi, Xiaolei Diao, Chuntao Li, and Hao Xu. 2022. Zinet: Linking chinese characters spanning three thousand years. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3061–3070. find
  10. Xiaolei Diao. 2022. Building a visual semantics aware object hierarchy. In Proceedings of the 31st International Joint Conference on Artificial Intelligence and the 25th European Conference on Artificial Intelligence, IJCAI-ECAI 2022. find
  11. Xiaolei Diao, Daqian Shi, Wei Cao, Ting Wang, Ruihua Qi, Chuntao Li, and Hao Xu. 2025. Oracle bone inscription image restoration via glyph extraction. npj Heritage Science, page 321. find
  12. Xiaolei Diao, Daqian Shi, Jian Li, Lida Shi, Mingzhe Yue, Ruihua Qi, Chuntao Li, and Hao Xu. 2023. Toward zero-shot character recognition: A gold standard dataset with radical-level annotations. In Proceedings of the 31st ACM International Conference on Multimedia, pages 6869–6877. find
  13. Xiaolei Diao, Zhihan Zhou, Lida Shi, Ting Wang, Ruihua Qi, Daqian Shi, and Hao Xu. 2026. Intechar: A unified oracle bone character list for ancient chinese language modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 211–219. find
  14. Fausto Giunchiglia, Mayukh Bagchi, and Xiaolei Diao. 2023. Aligning visual and lexical semantics. In International Conference on Information, pages 294– 302. find
  15. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, and Aobo Yang. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  16. Wang Guiyuan. 2023. A Study of Excavated Documents in China. Routledge, London. find
  17. HyoJung Han, Akiko Eriguchi, Haoran Xu, Hieu Hoang, Marine Carpuat, and Huda Khayrallah. 2025. Adapters for altering llm vocabularies: What languages benefit the most? In International Conference on Representation Learning, volume 2025, pages 68433–68459. find
  18. Xiaowei Han, Lizhen Xu, and Feng Qiao. 2018. Cnn-bilstm-crf model for term extraction in chinese corpus. In International Conference on Web Information Systems and Applications, pages 267–274. Springer. find
  19. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. Preprint, arXiv:1902.00751. find
  20. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  21. Q Jane. 2014. Ancient times table hidden in chinese bamboo strips. find
  22. Zongyuan Jiang, Jiapeng Wang, Jiahuan Cao, Xue Gao, and Lianwen Jin. 2023. Towards better translations from classical to modern chinese: A new dataset and a new method. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 387–399. Springer. find
  23. Teruno Kajiura, Shiho Takano, Tatsuya Hiraoka, and Kimio Kuramitsu. 2023. Vocabulary replacement in sentencepiece for domain adaptation. In Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, pages 645–652. find
  24. Seungduk Kim, Seungtaek Choi, and Myeongho Jeong. 2024. Efficient and effective vocabulary expansion towards multilingual large language models. arXiv preprint arXiv:2402.14714.
  25. Wenhao Li, Fanchao Qi, Maosong Sun, Xiaoyuan Yi, and Jiarui Zhang. 2021. Ccpm: A chinese classical poetry matching dataset. Preprint, arXiv:2106.01979. — [arXiv:2106.01979](https://arxiv.org/abs/2106.01979) find
  26. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4582–4597. Association for Co… find
  27. Xiaomeng Pan, Hongfei Wang, Teruaki Oka, and Mamoru Komachi. 2022. Zuo zhuan Ancient Chinese dataset for word sense disambiguation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Work… find
  28. Evgenia Rusak, Patrik Reizinger, Attila Juhos, Oliver Bringmann, Roland S. Zimmermann, and Wieland Brendel. 2024. Infonce: Identifying the gap between theory and practice. find
  29. Jiu Sha, Mengxiao Zhu, Chong Feng, and Yuming Shang. 2025. Veef-multi-llm: Effective vocabulary expansion and parameter efficient finetuning towards multilingual large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7963–7981. find
  30. Daqian Shi, Xiaolei Diao, Xu Chen, and Cédric M John. 2025a. Competitive distillation: A simple learning strategy for improving visual classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2981–2990. find
  31. Daqian Shi, Xiaolei Diao, Lida Shi, Hao Tang, Yang Chi, Chuntao Li, and Hao Xu. 2022a. Charformer: A glyph fusion based attentive framework for high-precision character image denoising. In Proceedings of the 30th ACM International Conference on Multimedia. find
  32. Daqian Shi, Xiaolei Diao, Hao Tang, Xiaomin Li, Hao Xing, and Hao Xu. 2022b. Rcrn: Real-world character image restoration network via skeleton extraction. In Proceedings of the 30th ACM International Conference on Multimedia. find
  33. Lida Shi, Fausto Giunchiglia, Hongda Zhang, Daqian Shi, Rui Song, Jian Li, Xiaolei Diao, Alan Zhao, and Hao Xu. 2025b. Learn from the best: A universal self-distillation approach with historical logits. Expert Systems with Applications, page 129340. find
  34. Juhee Son, Jiho Jin, Haneul Yoo, JinYeong Bak, Kyunghyun Cho, and Alice Oh. 2022. Translating hanja historical documents to contemporary korean and english. arXiv preprint arXiv:2205.10019. find
  35. Sho Takase, Ryokan Ri, Shun Kiyono, and Takuya Kato. 2025. Large vocabulary size improves large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 1015–1026. find
  36. Huishuang Tian, Kexin Yang, Dayiheng Liu, and Jiancheng Lv. 2021. Anchibert: A pre-trained model for ancient chinese language understanding and generation. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE. find
  37. D Wang and 1 others. 2022. Sikubert and sikuroberta: Construction and application research of pre training models for digital humanities in the complete library of four branches. In Lib. Forum, volume 42, pages 31–43. find
  38. Yuting Wei, Yuanxing Xu, Xinru Wei, Yangsimin Yangsimin, Yangfu Zhu, Yuqing Li, Di Liu, and Bin Wu. 2024. AC-EVAL: Evaluating Ancient Chinese language understanding in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1600–1617, Miami, Florida, US… find
  39. Shen Xu, editor. 1981. Shuowen Jiezi Zhu. Shanghai Ancient Books Publishing House, Shanghai. find
  40. Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. 2025. How can we effectively expand the vocabulary of llms with 0.01 gb of target language text? Computational Linguistics, pages 1–40. find
  41. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, and Haoze Sun. 2025a. Baichuan 2: Open large-scale language models. Preprint, arXiv:2309.10305.
  42. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and Junyang Lin. 2025b. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  43. Xinyu Yao, Mengdi Wang, Bo Chen, and Xiaobing Zhao. 2025. Wenyangpt: A large language model for classical chinese tasks. arXiv preprint arXiv:2504.20609. — [arXiv:2504.20609](https://arxiv.org/abs/2504.20609) find
  44. Yuan Yao, Qingxiu Dong, Jian Guan, Boxi Cao, Zhengyan Zhang, Chaojun Xiao, Xiaozhi Wang, Fanchao Qi, Junwei Bao, Jinran Nie, Zheni Zeng, Yuxian Gu, Kun Zhou, and Xuancheng Huan. 2022. Cuge: A chinese language understanding and generation evaluation benchmark. Preprint, arXiv:2112.13610. — [arXiv:2112.13610](https://arxiv.org/abs/2112.13610) find
  45. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, and Guanwei Zhang. 2025. Yi: Open foundation models by 01.ai. Preprint, arXiv:2403.04652.
  46. Peng Yu and Xin Wang. 2020. Bert-based named entity recognition in chinese twenty-four histories. In International Conference on Web Information Systems and Applications, pages 289–301. Springer. find
  47. Mingzhe Yue, Daqian Shi, Xiaolei Diao, Shuzhen Guo, Chuntao Li, and Hao Xu. 2025. Ancient character detection based on fine-grained density map. npj Heritage Science, 13(1):280. find
  48. Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, and 1 others. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. CoRR.
  49. Jundong Zhang, Songhua Yang, Jiangfeng Liu, and 1 others. 2024a. Aigc empowering the revitalization of ancient books on traditional chinese medicine: building the huang-di large language model. Libr Trib, 44:103–12. find
  50. Yixuan Zhang and Haonan Li. 2023. Can large language model comprehend Ancient Chinese? a preliminary test on ACLUE. In Proceedings of the Ancient Language Processing Workshop, pages 80–87, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. find
  51. Yuqing Zhang, Baoyi He, Yihan Chen, Hangqi Li, Han Yue, Shengyu Zhang, Huaiyong Dou, Junchi Yan, Zemin Liu, Yongquan Zhang, and 1 others. 2024b. Philogpt: A philology-oriented large language model for ancient chinese manuscripts with dunhuang as case study. In Proceedings of the 2024 Conference on E… find
  52. Zhixiao Zhao, Si Shen, Bin Li, and Xueliang Ma. 2024. Xunziallm. find
  53. Zhihan Zhou, Daqian Shi, Rui Song, Lida Shi, Xiaolei Diao, and Hao Xu. 2026. Ancientbench: Towards comprehensive evaluation on excavated and transmitted chinese corpora. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 35167– 35175. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-acse-ancient-character-semantic-aware/