Elamur Research⌕
☰
home / papers / 2026 / Neo-Classic: A Benchmark for Evaluating Linguistic-Aesthetic Reasoning in Classical Chinese Poetry

Neo-Classic: A Benchmark for Evaluating Linguistic-Aesthetic Reasoning in Classical Chinese Poetry

Paper · 2026 · Han Zhang, Zihan Gu, Zhiyuan Wang et al.

Summary

LLMs score very highly on existing Classical Chinese Poetry benchmarks such as WenMind, CCPM and ACP. Because canonical Tang and Song poems are almost certainly in pre-training data, those scores may reflect recall rather than genuine understanding of poetic rules. NEO-CLASSIC tests this by comparing model behaviour on historical poems with behaviour on 1,406 new works by contemporary experts. The new poems strictly follow the tonal, rhyme, parallelism and metrical rules of Regulated Verse (Lyu) and Ci, but their content does not appear in historical corpora. The benchmark uses five probe types, including authorship attribution, couplet matching and a sentence reordering task that requires planning at the level of the whole poem. Models such as Qwen3-Max, Gemini-3-Pro, DeepSeek-V3.2 and GPT-5-Pro were evaluated against human experts. On contemporary texts, model accuracy fell by roughly 20–50%, while expert accuracy stayed stable. Reordering accuracy was very low. The results suggest that current LLMs handle local form but lack robust global, hierarchical reasoning about poems, and that contamination-free test data is needed to measure this.

Key points

  • Corpus: 1,406 contemporary Lyu and Ci works by 30 consenting poets, curated with the SJTU Classical Chinese Culture Club. Code and data are at github.com/lyy0323/NEO-CLASSIC.
  • Memorization Gap: Accuracy on authorship and couplet-matching tasks drops by about 20–50% when moving from historical to contemporary poems. Human experts show little drop.
  • Sentence reordering: Standard models score 0–13%. Gemini-3-Pro reaches 36% with Expert-Level Guidance prompts, compared with 48% for human experts.
  • Error analysis on formal constraints:
    • Gemini-3-Pro scores 59% on rhyme validation and 55% on parallel-couplet identification.
    • Experts score 86% and 80% on the same checks.
    • Experts make more purely semantic ordering errors (26%), for example swapping the opening and closing lines.
  • Limitations: The corpus is small, the expert baseline comes only from university participants, and the analysis is purely behavioural, with no study of attention or other internal mechanisms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations🏛️Chinese Academy of SciencesBeijing · China246 works · 24,951 citations🎓University of Chinese Academy of SciencesBeijing · China188 works · 8,743 citations🔬Institute of Information EngineeringBeijing · China41 works · 1,485 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksLiterature and Literary Theory · 541 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,685 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Han Zhang, Zihan Gu, Zhiyuan Wang, Tianyi Ma, Jiacheng Lu, Xinyan Zhang, Yuhao Wei and Cheng Hua. 2026. Neo-Classic: A Benchmark for Evaluating Linguistic-Aesthetic Reasoning in Classical Chinese Poetry. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1266

@inproceedings{zhang2026neo,
  title = {{Neo-Classic: A Benchmark for Evaluating Linguistic-Aesthetic Reasoning in Classical Chinese Poetry}},
  author = {Han Zhang, Zihan Gu, Zhiyuan Wang, Tianyi Ma, Jiacheng Lu, Xinyan Zhang, Yuhao Wei and Cheng Hua},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1266},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1266}
}

References

33 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 01.AI. 2025. Yi: Open Foundation Models by 01.AI.
  2. Wafa Al Ghallabi, Ritesh Thawkar, Sara Ghaboura, Ketan Pravin More, Omkar Thawakar, Hisham Cholakkal, Salman Khan, and Rao Muhammad Anwer. 2025. Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natur…
  3. Danyang Cao, Cheng Cheng, and Guanmin Zhang. 2025. Compact and efficient language modeling for classical poetry: Generation and interpretation. 31(5):400. find
  4. Jiahuan Cao, Yang Liu, Yongxin Shi, Kai Ding, and Lianwen Jin. 2024. WenMind: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Classical Literature and Language Arts. Advances in Neural Information Processing Systems, 37:51358–51410. find
  5. Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. 2024. Art or artifice? large language models and the false promise zhiof creativity. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA. Association for… find
  6. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A Survey on Evaluation of Large Language Models. 15(3):39:1–39:45.
  7. Andong Chen, Lianzhang Lou, Kehai Chen, Xuefeng Bai, Yang Xiang, Muyun Yang, Tiejun Zhao, and Min Zhang. 2025. Benchmarking LLMs for Translating Classical Chinese Poetry: Evaluating Adequacy, Fluency, and Elegance. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Proces…
  8. DeepSeek-AI. 2025a. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  9. DeepSeek-AI. 2025b. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. Preprint, arXiv:2512.02556.
  10. Robert C. Gale, Alexandra C. Salem, Gerasimos Fergadiotis, and Steven Bedrick. 2023. Mixed Orthographic/Phonemic Language Modeling: Beyond Orthographically Restricted Transformers (BORT). In Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023), pages 212–225, Toronto, C… find
  11. Gemini Team. 2025a. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. Preprint, arXiv:2507.06261.
  12. Gemini Team. 2025b. Gemini 3 Pro Model Card. find
  13. Jinyi Hu and Maosong Sun. 2020. Generating Major Types of Chinese Classical Poetry in a Uniformed Framework. Preprint, arXiv:2003.11528. find
  14. Ilya Koziev. 2025. Automated Evaluation of Meter and Rhyme in Russian Generative and Human-Authored Poetry. Preprint, arXiv:2502.20931. — [arXiv:2502.20931](https://arxiv.org/abs/2502.20931) find
  15. Chalamalasetti Kranti and Sowmya Vajjala. 2025. MetricalARGS: A Taxonomy for Studying Metrical Poetry with LLMs. Preprint, arXiv:2510.08188. — [arXiv:2510.08188](https://arxiv.org/abs/2510.08188) find
  16. Jey Han Lau, Trevor Cohn, Timothy Baldwin, Julian Brooke, and Adam Hammond. 2018. Deep-speare: A Joint Neural Model of Poetic Language, Meter and Rhyme. Preprint, arXiv:1807.03491. find
  17. Wenhao Li, Fanchao Qi, Maosong Sun, Xiaoyuan Yi, and Jiarui Zhang. 2021. CCPM: A Chinese Classical Poetry Matching Dataset. Preprint, arXiv:2106.01979. — [arXiv:2106.01979](https://arxiv.org/abs/2106.01979) find
  18. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  19. Yang Liu, Lan Lan, Jiahuan Cao, Hiuyi Cheng, Kai Ding, and Lianwen Jin. 2025. Large-Scale Corpus Construction and Retrieval-Augmented Generation for Ancient Chinese Poetry: New Method and Data Insights. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 779–817. Associat… find
  20. Bolei Ma, Yina Yao, and Anna-Carolina Haensch. 2025a. Capabilities and Evaluation Biases of Large Language Models in Classical Chinese Poetry Generation: A Case Study on Tang Poetry. Preprint, arXiv:2510.15313. find
  21. Boxiang Ma, Ru Li, Wang Yuanlong, Hongye Tan, and Xiaoli Li. 2025b. Memorization ̸ = understanding: Do large language models have the ability of scenario cognition? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20758–20774, Suzhou, China. Associatio… find
  22. OpenAI. 2024. GPT-4o System Card.
  23. OpenAI. 2025. GPT-5 System Card. find
  24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  25. Andrei Popescu-Belis, Àlex R. Atrio, Bastien Bernath, Etienne Boisson, Teo Ferrari, Xavier Theimer-Lienhard, and Giorgos Vernikos. 2023. GPoeT: A Language Model Trained for Rhyme Generation on Synthetic Data. In Proceedings of the 7th Joint SIGHUM Workshop on Computational Linguistics for Cultural H… find
  26. Zhan Qu, Shuzhou Yuan, and Michael Färber. 2025. PoeTone: A Framework for Constrained Generation of Structured Chinese Songci with LLMs. Preprint, arXiv:2508.02515. — [arXiv:2508.02515](https://arxiv.org/abs/2508.02515) find
  27. Qwen Team. 2025. Qwen3 Technical Report. find
  28. Xinglong Song, Changlin Song, Haolu Yu, Yonghua Zhu, and Hong Yao. 2025. MixSong: Diverse and Strictly Formatted Chinese Poetry Generation. 24(4):37:1–37:16. find
  29. Melanie Walsh, Anna Preus, and Maria Antoniak. 2024. Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets. Preprint, arXiv:2406.18906. find
  30. Xunzi-LLM-of-Chinese-classics Team. 2023. Xuannziallm: A large language model for chinese classics. https://github.com/ Xunzi-LLM-of-Chinese-classics/XunziALLM. Open-source language model. ↗
  31. Chengyue Yu, Lei Zang, Jiaotuan Wang, Chenyi Zhuang, and Jinjie Gu. 2024. CharPoet: A Chinese Classical Poetry Generation System Based on Token-free LLM. Preprint, arXiv:2401.03512. find
  32. Yixuan Zhang and Haonan Li. 2023. Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE. Preprint, arXiv:2310.09550. find
  33. Shangqing Zhao, Yuhao Zhou, Yupei Ren, Zhe Chen, Chenghao Jia, Fang Zhe, Zhaogaung Long, Shu Liu, and Man Lan. 2025. Fùxì: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generation. Preprint, arXiv:2503.15837. — [arXiv:2503.15837](https://arxiv.org/abs/2503.15837) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-neo-classic-benchmark-evaluating-linguistic/