Elamur Research⌕
☰
home / papers / 2026 / Large-Scale Multimodal Knowledge Graph about Classical Chinese Poetry: Fine-grained Method and Comprehensive Evaluation

Large-Scale Multimodal Knowledge Graph about Classical Chinese Poetry: Fine-grained Method and Comprehensive Evaluation

Paper · 2026 · Shuo Wang, Qing Zhu, Yang Xiao et al.

Summary

Research on classical Chinese poetry has relied mostly on text. The only existing text-vision knowledge graph, PKG, covers imagery alone, includes no sound, and links to many web images that no longer exist. To fill this gap, the authors build CPMK, a multimodal knowledge graph that combines textual, visual and auditory knowledge.

The construction starts from an ontology derived from the data needs of recent poetry studies:

  • Text: poems and author information come from SouYun. Word and character meanings, stroke-order graphics and Pinyin/Zhuyin audio come from HanDian. A knowledge-augmentation step checks multiple sources against each other to handle textual variants and unreliable websites.
  • Images: pictures of poetic imagery are generated rather than scraped. Prompts are optimized first, and text-image alignment then removes weak pairs.

The authors test CPMK on four downstream tasks:

  • question answering
  • theme classification
  • poetry-image retrieval, using their new CPMK-based model KPIR
  • rigid-format poetry generation

The resource improves results in each case. It gives cultural NLP and multimodal work a sizeable open foundation.

Key points

  • Size: CPMK has 6,834,825 textual, 211,467 visual and 82,679 auditory nodes. It is released at github.com/Sure-aa/CPMK.
  • Motivation from PKG: 12,854 of 59,721 sampled PKG image URLs (21%) were broken.
  • Retrieval benchmarks: the authors created the first two benchmarks for this task, PI-Manual (70 hand-collected pairs) and PI-Generate (1,000 generated images).
  • Retrieval results: KPIR-0.4B, initialized from CN-CLIP, reaches t2i R@3 of 0.914 on PI-Manual versus 0.714 for the best baseline.
  • Generation and Pinyin coverage: adding 20,000 Pinyin-annotated poems to the 19,244 SongNet training entries raised rhyme MA-F1 from 81.66 to 84.41. The best Pinyin tool, Pypinyin, covers only 74.22% of CPMK's 50,769 Pinyin characters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 657 worksKnowledge graphs & ontologies · 192 works

Cite

Cite as Shuo Wang, Qing Zhu, Yang Xiao and Minglong Lei. 2026. Large-Scale Multimodal Knowledge Graph about Classical Chinese Poetry: Fine-grained Method and Comprehensive Evaluation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.720

@inproceedings{wang2026large,
  title = {{Large-Scale Multimodal Knowledge Graph about Classical Chinese Poetry: Fine-grained Method and Comprehensive Evaluation}},
  author = {Shuo Wang, Qing Zhu, Yang Xiao and Minglong Lei},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.720},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.720}
}

References

47 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  3. James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, and 1 others. 2023. Improving image generation with better captions. Technical report, OpenAI. find
  4. Jiahuan Cao, Yang Liu, Yongxin Shi, Kai Ding, and Lianwen Jin. 2024. Wenmind: A comprehensive benchmark for evaluating large language models in chinese classical literature and language arts. Advances in Neural Information Processing Systems, 37:51358–51410. find
  5. Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Qinghong Yang, and Ledell Wu. 2023. AltCLIP: Altering the language encoder in clip for extended language capabilities. In Findings of the Association for Computational Linguistics, pages 8666–8682. find
  6. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. Deepseek-r1: Incentivizing reasoning capability in ll…
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171–4186.
  8. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 320–335.
  9. Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. 2024. Lightrag: Simple and fast retrieval-augmented generation. arXiv preprint arXiv:2410.05779.
  10. Ming He, Yan Chen, Hong-Ke Zhao, Qi Liu, Le Wu, Yu Cui, Gui-Hua Zeng, and Gui-Quan Liu. 2023. Composing like an ancient chinese poet: Learn to generate rhythmic chinese poetry. Journal of Computer Science and Technology, 38(6):1272–1287. find
  11. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514–7528.
  12. Liang Hong, Wenjun Hou, and Lina Zhou. 2020. Knowpoetry: A knowledge service platform for tang poetry research based on domain-specific knowledge graph. Library Trends, 69:101–124. find
  13. IDEA-CCNL. 2024. Idea-ccnl ziya-llama-13b-v1.1. https://huggingface.co/IDEA-CCNL/ Ziya-LLaMA-13B-v1.1. ↗
  14. Jing Jiang, Yiran Ling, Binzhu Li, Pengxiang Li, Junming Piao, and Yu Zhang. 2024. Poetry2image: An iterative correction framework for images generated from chinese classical poetry. arXiv preprint arXiv:2407.06196. — [arXiv:2407.06196](https://arxiv.org/abs/2407.06196) find
  15. Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  16. Piji Li, Haisong Zhang, Xiaojiang Liu, and Shuming Shi. 2020. Rigid formats controlled text generation. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 742– 751. find
  17. Wenhao Li, Fanchao Qi, Maosong Sun, Xiaoyuan Yi, and Jiarui Zhang. 2021. Ccpm: A chinese classical poetry matching dataset. arXiv preprint arXiv:2106.01979. — [arXiv:2106.01979](https://arxiv.org/abs/2106.01979) find
  18. Yuqing Li, Yuxin Zhang, Bin Wu, Ji-Rong Wen, Ruihua Song, and Ting Bai. 2022. A multi-modal knowledge graph for classical chinese poetry. In Findings of the Association for Computational Linguistics, pages 2318–2326. find
  19. Lixin Liu, Xiaojun Wan, and Zongming Guo. 2018. Images2poem: Generating chinese poetry from image streams. In Proceedings of the 26th ACM international conference on Multimedia, pages 1967–1975. find
  20. Yang Liu, Lan Lan, Jiahuan Cao, Hiuyi Cheng, Kai Ding, and Lianwen Jin. 2025. Large-scale corpus construction and retrieval-augmented generation for ancient chinese poetry: New method and data insights. In Findings of the Association for Computational Linguistics, pages 779–817. find
  21. Yulong Liu, Guibo Zhu, Bin Zhu, Qi Song, Guojing Ge, Haoran Chen, GuanHui Qiao, Ru Peng, Lingxiang Wu, and Jinqiao Wang. 2022. Taisu: A 166m large-scale high-quality dataset for chinese vision-language pre-training. Advances in Neural Information Processing Systems, 35:16705–16717. find
  22. Yusen Liu, Dayiheng Liu, Jiancheng Lv, and Yongsheng Sang. 2020. Generating chinese poetry from images via concrete and abstract information. In 2020 International Joint Conference on Neural Networks, pages 1–8. lxneng. 2025. xpinyin. project/xpinyin/. https://pypi.org/lxyu. 2016. pinyin. https://py… ↗
  23. Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. 2022. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the 39th International Conference on Machine Learni… find
  24. PeterH0323. 2024. Peterh0323 ancient-chat-https://github.com/PeterH0323/ llm. ancient-chat-llm. find
  25. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125.
  26. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695. find
  27. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, and 1 others. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information proce…
  28. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179… find
  29. VMIJUNV. 2023. Chinese-poetry-and-prose. https://github.com/VMIJUNV/ Chinese-poetry-and-prose. ↗
  30. Xueyao Wan and Hang Yu. 2025. Mmgraphrag: Bridging vision and language with interpretable multimodal knowledge graphs. arXiv preprint arXiv:2507.20804. — [arXiv:2507.20804](https://arxiv.org/abs/2507.20804) find
  31. Chengyu Wang, Zhongjie Duan, Bingyan Liu, Xinyi Zou, Cen Chen, Kui Jia, and Jun Huang. 2024. PAI-diffusion: Constructing and serving a family of open Chinese diffusion models for text-to-image synthesis on the cloud. In Proceedings of the 62nd Annual Meeting of the Association for Computational Ling… find
  32. Qing Wang, Weiping Liu, Xiumei Wang, Xinghong Chen, Guannan Chen, and Qingxiang Wu. 2023. A spatial – temporal graph model for pronunciation feature prediction of chinese poetry. IEEE Transactions on Neural Networks and Learning Systems, 34(12):10294–10308. find
  33. Changlong Yu, Weiqi Wang, Xin Liu, Jiaxin Bai, Yangqiu Song, Zheng Li, Yifan Gao, Tianyu Cao, and Bing Yin. 2023. FolkScope: Intention knowledge graph construction for E-commerce commonsense discovery. In Findings of the Association for Computational Linguistics, pages 1173–1191. find
  34. Qingwang Wang, Chaohui Li, Yi Liu, Qiubai Zhu, Jian Song, and Tao Shen. 2025. An adaptive framework embedded with llm for knowledge graph construction. IEEE Transactions on Multimedia, 27:2912– 2923. find
  35. Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. 2024. Long-clip: Unlocking the long-text capability of clip. In European Conference on Computer Vision, pages 310–325. find
  36. Yuting Wei, Linmei Hu, Yangfu Zhu, Jiaqi Zhao, and Bin Wu. 2024. Knowledge-guided transformer for joint theme and emotion classification of chinese classical poetry. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:4783–4794. find
  37. Werneror. 2017. Poetry. Werneror/Poetry. https://github.com/ ↗
  38. Xiaojun Wu, Dixiang Zhang, Ruyi Gan, Junyu Lu, Ziwei Wu, Renliang Sun, Jiaxing Zhang, Pingjian Zhang, and Yan Song. 2024. Taiyi-diffusion-xl: advancing bilingual text-to-image generation with large vision-language model support. arXiv preprint arXiv:2401.14688. — [arXiv:2401.14688](https://arxiv.org/abs/2401.14688) find
  39. Chunyu Xie, Heng Cai, Jincheng Li, Fanjing Kong, Xiaoyu Wu, Jianfei Song, Henrique Morimitsu, Lin Yao, Dexin Wang, Xiangzheng Zhang, and 1 others. 2023. CCMB: A large-scale chinese cross-modal benchmark. In Proceedings of the 31st ACM International Conference on Multimedia, pages 4219–4227. find
  40. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, and 1 others. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.
  41. An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, and Chang Zhou. 2022. Chinese clip: Contrastive vision-language pretraining in chinese. arXiv preprint arXiv:2211.01335. — [arXiv:2211.01335](https://arxiv.org/abs/2211.01335) find
  42. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 43 others. 2024. Qwen2 technical report. arXiv preprint arXi… find
  43. Fulong Ye, Guang Liu, Xinya Wu, and Ledell Wu. 2024. Altdiffusion: A multilingual text-to-image diffusion model. In Proceedings of the AAAI conference on artificial intelligence, pages 6648–6656. find
  44. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, and 1 others. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652.
  45. Jiaxing Zhang, Ruyi Gan, Junjie Wang, Yuxiang Zhang, Lin Zhang, Ping Yang, Xinyu Gao, Ziwei Wu, Xiaoqun Dong, Junqing He, and 1 others. 2022. Fengshenbang 1.0: Being the foundation of chinese cognitive intelligence. arXiv preprint arXiv:2209.02970. — [arXiv:2209.02970](https://arxiv.org/abs/2209.02970) find
  46. Jiaqi Zhao, Ting Bai, Yuting Wei, and Bin Wu. 2022. Poetrybert: Pre-training with sememe knowledge for classical chinese poetry. In International conference on data mining and big data, pages 369–384. find
  47. Jialong Zuo, Haoyou Deng, Hanyu Zhou, Jiaxin Zhu, Yicheng Zhang, Yiwei Zhang, Yongxin Yan, Kaixing Huang, Weisen Chen, Yongtai Deng, Rui Jin, Nong Sang, and Changxin Gao. 2025. Is nano banana pro a low-level vision all-rounder? a comprehensive evaluation on 14 tasks and 40 datasets. arXiv preprint a… — [arXiv:2512.15110](https://arxiv.org/abs/2512.15110) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-large-scale-multimodal-knowledge-graph/