Elamur Research⌕
☰
home / papers / 2026 / Poller: Are LLMs Suitable for Evaluating Poetry Understanding Task?

Poller: Are LLMs Suitable for Evaluating Poetry Understanding Task?

Paper · 2026 · Shanshan Wang, Derek F. Wong, Jingming Yao et al.

Summary

Standard automatic metrics do not capture the free form and figurative nature of modern Chinese poetry, and expert human judgment is reliable but too costly to scale. The authors test whether LLMs can grade machine interpretations of poems. They build a benchmark of 40 unpublished modern Chinese poems by five professional poets. GPT-4 writes interpretations of each poem across eight dimensions, and the poets themselves score those interpretations to provide the human reference.

Off-the-shelf LLM judges score far from the human reference. The authors therefore propose Poller (Poetry LLM Evaluator), which asks the judging model to role-play the poem's author. The prompt supplies the poet's background, the poet's own views on writing, and comments from professional critics. Several prompt variants (P1–P4) build on a plain baseline (P0). Across multiple LLMs, Poller substantially lowers the absolute error against human scores.

This makes LLM-assisted evaluation a practical option for poetry tasks. The authors note that the method depends on having background information about the author.

Key points

  • Data: 40 unpublished modern Chinese poems from 5 poets, with 8 poems by a single poet. Human judges rated GPT-4's interpretations as more than 76% accurate.
  • Eight scoring dimensions: content, language, imagery, rhetorical techniques, rhythm, defamiliarization, modernity, and thought & emotion. Each dimension is scored from 0 to 100, and the judge must justify its score.
  • Average error reduction versus baseline: 57.37% for GPT-4-Turbo and 46.95% for DeepSeek-R1.
  • Per-dimension error reduction: up to 94.55% for rhetorical techniques and 89.53% for defamiliarization.
  • Best prompt variant: P3, which requires the model to state its assumed role before scoring, worked best in nearly every case. The one exception was Qwen2.5-Turbo on content, where P4 was best. Recommended pairings include DeepSeek-R1 for rhetorical techniques, GPT-4-Turbo for defamiliarization and rhythm, and Claude3.5-sonnet for language.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MacauMacao · China33 works · 2,223 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Shanshan Wang, Derek F. Wong, Jingming Yao and Lidia S. Chao. 2026. Poller: Are LLMs Suitable for Evaluating Poetry Understanding Task?. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.472

@inproceedings{wang2026poller,
  title = {{Poller: Are LLMs Suitable for Evaluating Poetry Understanding Task?}},
  author = {Shanshan Wang, Derek F. Wong, Jingming Yao and Lidia S. Chao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.472},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.472}
}

References

65 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Manex Agirrezabal and Hugo Gonçalo Oliveira. 2024. Zero-shot metrical poetry generation with open language models: a quantitative analysis. find
  3. Manex Agirrezabal, Hugo Gonçalo Oliveira, and Aitor Ormazabal. 2023. Erato: Automatizing poetry evaluation. In EPIA Conference on Artificial Intelligence, pages 3–14. Springer. find
  4. D Antar. 2023. The effectiveness of using chatgpt4 in creative writing in arabic: Poetry and short story as a model. find
  5. Anthropic. 2024. Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 sonnet. find
  6. Abdul Ghafoor Awan and Yasmin Khalida. 2015. New trends in modern poetry. Journal of Literature, Languages and Linguistics, 13:63–72. find
  7. Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schmidt. 2023. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use. arXiv preprint arXiv:2308.06595. — [arXiv:2308.06595](https://arxiv.org/abs/2308.06595) find
  8. J. Brodsky. 1999. Witness and Pleasure. witness and pleasure. find
  9. Tuhin Chakrabarty, Arkadiy Saakyan, and Smaranda Muresan. 2021. Don’t go far off: An empirical study on neural poetry translation. arXiv preprint arXiv:2109.02972. find
  10. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201.
  11. Yina Chang. 2012. A brief discussion on cutting-edge issues in contemporary poetry research. Northern Literature Monthly. find
  12. Bo Cheng. 2005. The special ecology of modernity in new poetry–a study on the poets of southwest associated university. Journal of the School of Liberal Arts of Nanjing Normal University, (4):69–76. find
  13. Cheng-Han Chiang and Hung-Yi Lee. 2023a. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631. find
  14. Cheng-Han Chiang and Hung-yi Lee. 2023b. A closer look into using large language models for automatic evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8928– 8942. Association for Computational Linguistics. find
  15. Zhumin Chu, Qingyao Ai, Yiteng Tu, Haitao Li, and Yiqun Liu. 2024. Pre: A peer review based large language model evaluator. arXiv preprint arXiv:2401.15641. — [arXiv:2401.15641](https://arxiv.org/abs/2401.15641) find
  16. John Crowe Ransom. 1938. The world’s body (baton rouge). find
  17. Jonathan Culler. 2017. Theory of the lyric. Nordisk poesi, 2(2):119–133. find
  18. Zekun Deng, Hao Yang, and Jun Wang. 2024. Can ai write classical chinese poetry like humans? an empirical study inspired by turing test. arXiv preprint arXiv:2401.04952. — [arXiv:2401.04952](https://arxiv.org/abs/2401.04952) find
  19. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  20. James Hutson and Ana Schnellmann. 2023. The poetry of prompts: the collaborative role of generative artificial intelligence in the creation of poetry and the anxiety of machine influence. Global Journal of Computer Science and Technology, 23(D1):1–14. find
  21. Yongjun Jiang. 2012. The "unspeakable" in the perceptual effect of poetry - an analysis of the "unspeakable" meaning of feeling. foreign language, 28(1):4. find
  22. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2023. Benchmarking cognitive biases in large language models as evaluators. arXiv preprint arXiv:2309.17012. find
  23. Tian Lan, Jiang Li, Yemin Wang, Xu Liu, Xiangdong Su, and Guanglai Gao. 2025a. F 2 Bench: An open-ended fairness evaluation benchmark for LLMs with factuality considerations. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2031–2046. Association for C… find
  24. Tian Lan, Xiangdong Su, Xu Liu, Ruirui Wang, Ke Chang, Jiang Li, and Guanglai Gao. 2025b. McBE: A multi-task Chinese bias evaluation benchmark for large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6033–6056. find
  25. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579.
  26. Jiang Li, Tian Lan, Shanshan Wang, Dongxing Zhang, Dianqing Lin, Guanglai Gao, Derek F Wong, and Xiangdong Su. 2026. Who wrote this line? evaluating the detection of llm-generated classical chinese poetry. arXiv preprint arXiv:2604.10101. find
  27. Dianqing Lin, Aruukhan, Hongxu Hou, Shuo Sun, Wei Chen, Yichen Yang, and Guodong Shi. 2025. Can large language models translate unseen languages in underrepresented scripts? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 23137–23150. Association for… find
  28. Yen-Ting Lin and Yun-Nung Chen. 2023. Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. arXiv preprint arXiv:2305.13711.
  29. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522. Association for Computational Linguistics.
  30. Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. 2024. Llms as narcissistic evaluators: When ego inflates evaluation scores. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12688–12701. find
  31. Ruli Manurung, Graeme Ritchie, and Henry Thompson. 2012. Using genetic algorithms to create meaningful poetic text. Journal of Experimental & Theoretical Artificial Intelligence, 24(1):43–64. find
  32. Brian McGrath. 2018. Understating poetry. New Literary History, 49(3):289–308. find
  33. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252. Association for Computational Linguistics.
  34. Arjun Panickssery, Samuel R Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. arXiv preprint arXiv:2404.13076. find
  35. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics.
  36. Robert B Pierce. 2003. Defining" poetry". Philosophy and Literature, 27(1):151–163. find
  37. Maurilio De Araujo Possi, Alcione De Paiva Oliveira, Alexandra Moreira, and Lucas Mucida Costa. 2023. Carmen: A method for automatic evaluation of poems. In 2023 5th International Conference on Natural Language Processing (ICNLP), pages 244–247. IEEE. find
  38. Eshrag Ali Refaee. 2023. Okaz: A deep-learning-based system for automatic arabic poem generation. In 2023 3rd International Conference on Computing and Information Technology (ICCIT), pages 396–403. IEEE. find
  39. Philip Resnik and Jimmy Lin. 2010. Evaluation of nlp systems. The handbook of computational linguistics and natural language processing, pages 271–295. find
  40. Hanwen Shen, Ting Ying, Jiajie Lu, and Shanshan Wang. 2026. Preconditioned test-time adaptation for out-of-distribution debiasing in narrative generation. arXiv preprint arXiv:2603.13683.
  41. Brian Phillips Skerratt. 2013. Form and Transformation in Modern Chinese Poetry and Poetics. Ph.D. thesis. find
  42. Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024. Finesure: Fine-grained summarization evaluation using llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 906–922.
  43. Wai Lei Song, Haoyun Xu, Derek F. Wong, Runzhe Zhan, Lidia S. Chao, and Shanshan Wang. 2023. Towards zero-shot multilingual poetry translation. In Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 324–335. Asia-Pacific Association for Machine Translation. find
  44. Shuting Wang. 2006a. Giving shape to emotion: The theory and practice of rhythm and imagery in new poetry (1917-1937). World Literature Review. find
  45. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? In Chinese computational linguistics: 18th China national conference, CCL 2019, Kunming, China, October 18–20, 2019, proceedings 18, pages 194– 206. Springer.
  46. Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, and 1 others. 2023b. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087.
  47. Allen Tate. 1940. Understanding modern poetry. The English Journal, 29(4):263–274. find
  48. Maria Virvou, George A Tsihrintzis, Dionisios N Sotiropouloss, Konstantina Chrysafiadi, Evangelos Sakkopoulos, and Evangelia-Aikaterini Tsichrintzi. 2023. Chatgpt in artificial intelligence-empowered e-learning for cultural heritage: The case of lyrics and poems. In 2023 14th International Conferenc… find
  49. Yuchun Wang. 2006b. Moruo guo’s view on literary translation. Moruo Guo Journal, page 5. find
  50. Yunshu Xi. 2019. Two characteristics of new poetry language—talking about the creation of new poetry part three. Masterpiece Appreciation: Appreciation Edition (First Decade), (8):6. find
  51. Xilie Xu, Keyi Kong, Ning Liu, Lizhen Cui, Di Wang, Jingfeng Zhang, and Mohan Kankanhalli. 2023. An llm can fool itself: A prompt-based adversarial attack. arXiv preprint arXiv:2310.13345. — [arXiv:2310.13345](https://arxiv.org/abs/2310.13345) find
  52. Melanie Walsh, Anna Preus, and Maria Antoniak. 2024. Sonnet or not, bot? poetry evaluation for large models and datasets. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15568– 15603. find
  53. Rui Yan. 2016. I, poet:automatic poetry composition through recurrent neural networks with iterative polishing schema. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, page 2238–2244. find
  54. Guangming Wang. 1998. Reflections on the ontology of new chinese poetry. Chinese Social Sciences, (4):16. find
  55. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  56. Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. Is ChatGPT a good NLG evaluator? a preliminary study. In Proceedings of the 4th New Frontiers in Summarization Workshop, pages 1–11. Association for Computational Linguistics.
  57. Mi Wang. 2019. Study on the relationship between external form and internal emotion in the rhythm of new poetry (1917-1949). Ph.D. thesis, Sichuan Academy of Social Sciences. find
  58. Shanshan Wang, Derek Wong, Jingming Yao, and Lidia Chao. 2024. What is the best way for chatgpt to translate poetry? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14025– 14043. find
  59. Shanshan Wang, Derek F. Wong, Jingming Yao, and Lidia S. Chao. 2026. Can ChatGPT really understand Modern Chinese poetry? In Findings of the Association for Computational Linguistics: EACL 2026, pages 4152–4162. Association for Computational Linguistics.
  60. Shanshan Wang, Junchao Wu, Fengying Ye, Derek F. Wong, Jingming Yao, and Lidia S. Chao. 2025. Benchmarking the detection of LLMs-generated Modern Chinese poetry. In Findings of the Association for Computational Linguistics: EMNLP 2025, Fengying Ye, Shanshan Wang, Lidia S Chao, and Derek F Wong. 2025… find
  61. Xiaoyuan Yi, Maosong Sun, Ruoyu Li, and Wenhao Li. 2018. Automatic poetry generation with mutual reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3143–3153. Association for Computational Linguistics. find
  62. Haizhang Yu. 2001. About the changes and analysis of new poetry expressions. writing, 000(011):3–5. find
  63. Di Zang. 2007. The literary politics of "poetic" - on the traces and limits of "poetic" in the practice of chinese new poetry. Comments on New Poetry, pages 13, 15–16, 19, 30. find
  64. Taozhou Zhang. 2022. Problems in reading new poetry from the perspective of "interpretation of poetics". literary research, (3):13. find
  65. Xingxing Zhang and Mirella Lapata. 2014. Chinese poetry generation with recurrent neural networks. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 670–680.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-poller-are-llms-suitable-evaluating/