Elamur Research⌕
☰
home / papers / 2026 / Can ChatGPT Really Understand Modern Chinese Poetry?

Can ChatGPT Really Understand Modern Chinese Poetry?

Paper · 2026 · Shanshan Wang, Derek F. Wong, Jingming Yao et al.

Summary

Prior work on ChatGPT and poetry has looked at generation, translation and detection outputs. This paper asks a more basic question: does the model actually comprehend modern Chinese verse? The authors propose ECUMP (Evaluation of ChatGPT's Understanding of Modern Poetry), a four-step framework:

  1. Define interpretation dimensions with input from professional poets.
  2. Design a structured prompt.
  3. Have the model interpret the poems.
  4. Have the poems' own authors grade the interpretations.

The authors argue that the poet's intent is the most defensible reference, since readers' interpretations of poetry vary widely. GPT-4 interpreted 48 newly written poems, which had no prior published analysis. Across most dimensions, the poets judged the interpretations to match their intentions in more than 73% of cases. Identifying the most poetic line was the weakest skill. When LLMs were used as automatic judges, their scores diverged sharply from the poets' scores. This matters for the field: LLMs can help readers approach modern poetry, but evaluating such understanding still needs expert humans.

Key points

  • Dimensions: The framework covers five dimensions: a poem's content, the methods it uses for expression, the thought and emotion it conveys, its modernity, and its poeticity. Expression methods splits into language, imagery, rhetorical techniques, rhythm and defamiliarization. Poeticity was tested by asking the model to pick the single most poetic sentence.
  • Dataset: Six poets each supplied 8 recent poems, giving 48 poems with 147 stanzas and 750 lines. These split into 40 Com-Poetry poems (common types, five poets) and 8 Spe-Poetry poems (classical imagery and historical figures, one poet). The model was GPT-4 (gpt-4-0125) at temperature 0.5 and top_p 1.0.
  • Scoring: Poets scored eight sub-dimensions from 0 to 100. Poeticity was scored only as 0, 50 or 100.
    • For Com-Poems, the best scores were imagery (81.18) and content (80.33).
    • For Spe-Poems, rhetorical techniques scored highest (88.75) and language lowest (73.75).
  • Poeticity results: The poets agreed with GPT-4's chosen line for 17 of 40 Com-Poems and 3 of 8 Spe-Poems. In 2 and 1 cases respectively, the chosen line was judged to lack poeticity entirely.
  • LLMs as judges: o1-preview, GPT-4o, Qwen2.5-Plus and Qwen2.5-Max were tested as evaluators. They gave inflated scores of roughly 81–97, far from the poets' ratings, so they are not yet reliable judges of poetic understanding.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksLiterature and Literary Theory · 519 works

Cite

Cite as Shanshan Wang, Derek F. Wong, Jingming Yao and Lidia S. Chao. 2026. Can ChatGPT Really Understand Modern Chinese Poetry?. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.216

@inproceedings{wang2026can,
  title = {{Can ChatGPT Really Understand Modern Chinese Poetry?}},
  author = {Shanshan Wang, Derek F. Wong, Jingming Yao and Lidia S. Chao},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.216},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.216}
}

References

47 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. D Antar. 2023. The effectiveness of using chatgpt4 in creative writing in arabic: Poetry and short story as a model. find
  3. J. Brodsky. 1999. Witness and Pleasure. witness and pleasure. find
  4. Tuhin Chakrabarty, Arkadiy Saakyan, and Smaranda Muresan. 2021. Don’t go far off: An empirical study on neural poetry translation. arXiv preprint arXiv:2109.02972. find
  5. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201.
  6. Yina Chang. 2012. A brief discussion on cutting-edge issues in contemporary poetry research. Northern Literature Monthly. find
  7. Sreejato Chatterjee, Linh Tran, Quoc Duy Nguyen, Roni Kirson, Drue Hamlin, Harvest Aquino, Hanjia Lyu, Jiebo Luo, and Timothy Dye. 2025. Assessing historical structural oppression worldwide via rule-guided prompting of large language models. arXiv preprint arXiv:2509.15216. — [arXiv:2509.15216](https://arxiv.org/abs/2509.15216) find
  8. Bo Cheng. 2005. The special ecology of modernity in new poetry–a study on the poets of southwest associated university. Journal of the School of Liberal Arts of Nanjing Normal University, (4):69–76. find
  9. Cheng-Han Chiang and Hung-Yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631. find
  10. Zhumin Chu, Qingyao Ai, Yiteng Tu, Haitao Li, and Yiqun Liu. 2024. Pre: A peer review based large language model evaluator. arXiv preprint arXiv:2401.15641. — [arXiv:2401.15641](https://arxiv.org/abs/2401.15641) find
  11. Zekun Deng, Hao Yang, and Jun Wang. 2024. Can ai write classical chinese poetry like humans? an empirical study inspired by turing test. arXiv preprint arXiv:2401.04952. — [arXiv:2401.04952](https://arxiv.org/abs/2401.04952) find
  12. Yuan Gao, Ruili Wang, and Feng Hou. 2023. How to design translation prompts for chatgpt: An empirical study. arXiv e-prints, pages arXiv–2304. find
  13. Dmitriy Genzel, Jakob Uszkoreit, and Franz Och. 2010. “poetic” statistical machine translation: Rhyme and meter. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 158–166, Cambridge, MA. Association for Computational Linguistics. find
  14. Marjan Ghazvininejad, Yejin Choi, and Kevin Knight. 2018. Neural poetry translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 67–71. find
  15. James Hutson and Ana Schnellmann. 2023. The poetry of prompts: the collaborative role of generative artificial intelligence in the creation of poetry and the anxiety of machine influence. Global Journal of Computer Science and Technology, 23(D1):1–14. find
  16. Yongjun Jiang. 2012. The "unspeakable" in the perceptual effect of poetry - an analysis of the "unspeakable" meaning of feeling. foreign language, 28(1):4. find
  17. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.08745, 1(10).
  18. Tian Lan, Jiang Li, Yemin Wang, Xu Liu, Xiangdong Su, and Guanglai Gao. 2025a. F 2 Bench: An open-ended fairness evaluation benchmark for LLMs with factuality considerations. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2031–2046. Association for C… find
  19. Tian Lan, Xiangdong Su, Xu Liu, Ruirui Wang, Ke Chang, Jiang Li, and Guanglai Gao. 2025b. McBE: A multi-task Chinese bias evaluation benchmark for large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6033–6056. find
  20. Dianqing Lin, Aruukhan, Hongxu Hou, Shuo Sun, Wei Chen, Yichen Yang, and Guodong Shi. 2025. Can large language models translate unseen languages in underrepresented scripts? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 23137–23150. Association for… find
  21. Yen-Ting Lin and Yun-Nung Chen. 2023. Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. arXiv preprint arXiv:2305.13711.
  22. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522. Association for Computational Linguistics.
  23. Quanming Long. 2005. Modernity of Chinese New Poetry. Modernity of Chinese New Poetry. find
  24. Hanjia Lyu, Jiebo Luo, Jian Kang, and Allison Koenecke. 2025. Characterizing bias: Benchmarking large language models in simplified versus traditional chinese. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, page 2815–2846. Association for Computing Machinery… find
  25. Hisar Manurung. 2004. An evolutionary algorithm approach to poetry generation. find
  26. Brian McGrath. 2018. Understating poetry. New Literary History, 49(3):289–308. find
  27. Qingwei Mo. 2009. Emotional classification of new poetry and its cultural roots. Management Observation, (13):1. find
  28. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252.
  29. Robert B Pierce. 2003. Defining" poetry". Philosophy and Literature, 27(1):151–163. find
  30. Weihong Qi, Hanjia Lyu, and Jiebo Luo. 2025. Representation bias in political sample simulations with large language models. In Companion Proceedings of the ACM on Web Conference 2025, page 1264–1267, New York, NY, USA. Association for Computing Machinery. find
  31. Eshrag Ali Refaee. 2023. Okaz: A deep-learning-based system for automatic arabic poem generation. In 2023 3rd International Conference on Computing and Information Technology (ICCIT), pages 396–403. IEEE. find
  32. Hanwen Shen and Ting Ying. 2025. Measuring information distortion in hierarchical ultra long novel generation: The optimal expansion ratio. arXiv preprint arXiv:2505.12572. — [arXiv:2505.12572](https://arxiv.org/abs/2505.12572) find
  33. Wai Lei Song, Haoyun Xu, Derek F. Wong, Runzhe Zhan, Lidia S. Chao, and Shanshan Wang. 2023. Towards zero-shot multilingual poetry translation. In Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 324–335. Asia-Pacific Association for Machine Translation. find
  34. Allen Tate. 1940. Understanding modern poetry. The English Journal, 29(4):263–274. find
  35. Tim Van de Cruys. 2020. Automatic poetry generation from prosaic text. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 2471–2480. find
  36. Maria Virvou, George A Tsihrintzis, Dionisios N Sotiropouloss, Konstantina Chrysafiadi, Evangelos Sakkopoulos, and Evangelia-Aikaterini Tsichrintzi. 2023. Chatgpt in artificial intelligence-empowered e-learning for cultural heritage: The case of lyrics and poems. In 2023 14th International Conferenc… find
  37. Guangming Wang. 1998. Reflections on the ontology of new chinese poetry. Chinese Social Sciences, (4):16. find
  38. Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048.
  39. Mi Wang. 2019. Study on the relationship between external form and internal emotion in the rhythm of new poetry (1917-1949). Ph.D. thesis, Sichuan Academy of Social Sciences. find
  40. Shanshan Wang, Derek Wong, Jingming Yao, and Lidia Chao. 2024. What is the best way for ChatGPT to translate poetry? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14025– 14043. find
  41. Shanshan Wang, Junchao Wu, Fengying Ye, Derek F. Wong, Jingming Yao, and Lidia S. Chao. 2025. Benchmarking the detection of LLMs-generated Modern Chinese poetry. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 9533–9552. Association for Computational Linguistics. theo… find
  42. Shuting Wang. 2006. Giving shape to emotion: The theory and practice of rhythm and imagery in new poetry (1917-1937). World Literature Review. find
  43. Yunshu Xi. 2019. Two characteristics of new poetry language—talking about the creation of new poetry part three. Masterpiece Appreciation: Appreciation Edition (First Decade), (8):6. find
  44. Fengying Ye, Shanshan Wang, Lidia S Chao, and Derek F Wong. 2025. Unveiling llms’ metaphorical understanding: Exploring conceptual irrelevance, context leveraging and syntactic influence. arXiv preprint arXiv:2510.04120. — [arXiv:2510.04120](https://arxiv.org/abs/2510.04120) find
  45. Michelle Yeh. 1990. A new orientation to poetry: The transition from traditional to modern. Chinese Literature: Essays, Articles, Reviews (CLEAR), 12:83–105. find
  46. Xiaoyuan Yi, Maosong Sun, Ruoyu Li, and Wenhao Li. 2018. Automatic poetry generation with mutual reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3143–3153. find
  47. Haizhang Yu. 2001. About the changes and analysis of new poetry expressions. writing, 000(011):3–5. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-can-chatgpt-really-understand-modern/