Elamur Research⌕
☰
home / papers / 2026 / Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese

Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese

Paper · 2026 · Xing Yue, Yongliang Shen and Weiming Lu

Summary

Most LLM research looks at meaning and spelling and largely ignores sound. Existing phonology tests have two weaknesses: models can pass them by memorizing pronunciations, or the tasks mix in skills such as semantics. The authors define phonological understanding as using sound knowledge flexibly in new situations rather than just recalling it. They set two design criteria: tasks must be memorization-proof, and they must be solvable through sound rather than meaning.

They chose Chinese because its logographic script gives fewer spelling-based hints about pronunciation. They built Phun-Bench around homophony, rhyme and phonetic similarity, using an eight-level similarity scale computed from pinyin initials, finals and tones.

A preliminary char2pinyin test shows that LLMs recall pronunciations better than humans do. On the benchmark tasks, however, the same models generally fall behind human intuition. Larger models and reasoning modes narrow this gap. The authors hypothesize that LLMs work by "reasoning via the sound layer," for example by spelling words out in pinyin. Audio models largely follow the same text-centred route. The benchmark separates knowing pronunciations from actually using them.

Key points

  • Tasks: Phun-Bench has four tasks:
    • Homophone Recall
    • Contextual Homophone Recognition
    • Rhyming Sentence Generation (single- and double-rhyme)
    • Similarity Comparison, tested under several perturbation types
  • Verification test: The char2pinyin check used 5,345 characters that each have only one pronunciation, taken from the 2013 List of Commonly-Used Standard Chinese Characters. LLMs generally outperformed humans on it.
  • Models evaluated: Llama 3.2-3B, Qwen3-8B, GLM4-9B, Qwen2.5-72B, DeepSeek-V3/R1, GPT-4o and QwQ-32B. Audio models included Qwen3-Omni-Flash. A scaling study of the full Qwen3 family found steady gains with size and from thinking mode. Double-rhyme generation stayed difficult even for the largest model.
  • Data sources: The data was built with jieba and pypinyin and corrected by hand. It draws on chinese-idiom-db, ChineseLyrics, news2016zh and DuanzAI.
  • Limitations: The benchmark covers Chinese only, and the human baseline is small (participants were paid 80 CNY for about 70 minutes). Data and code are released on GitHub (xing-stellus-yue/Phun-Bench).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Xing Yue, Yongliang Shen and Weiming Lu. 2026. Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1041

@inproceedings{yue2026phun,
  title = {{Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese}},
  author = {Xing Yue, Yongliang Shen and Weiming Lu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1041},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1041}
}

References

65 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John R. Anderson, Michael Matessa, and Christian Lebiere. 1997. Act-R: A Theory of Higher Level Cognition and Its Relation to Visual Attention. Human-Computer Interaction, 12(4):439–462. find
  2. Jonas Belouadi and Steffen Eger. 2023. ByGPT5: End-to-end style-conditioned poetry generation with token-free language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7364–7381, Toronto, Canada. Association for Computa… find
  3. Benjamin S Bloom, Max D Engelhart, Edward J Furst, Walker H Hill, and David R Krathwohl. 1956. Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain. Longman, New York. find
  4. Yang Chen, Chong Yang, Tu Hu, Xinhao Chen, Man Lan, Li Cai, Xinlin Zhuang, Xuan Lin, Xin Lu, and Aimin Zhou. 2024a. Are U a joke master? pun generation via multi-stage curriculum learning towards a humor LLM. In Findings of the Association for Computational Linguistics: ACL 2024, pages 878–890, Bang… find
  5. Yanran Chen, Hannes Gröner, Sina Zarrieß, and Steffen Eger. 2024b. Evaluating diversity in automatic poetry generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19671–19692, Miami, Florida, USA. Association for Computational Linguistics. find
  6. Yuyan Chen, Zhixu Li, Jiaqing Liang, Yanghua Xiao, Bang Liu, and Yunwen Chen. 2023. Can Pre-trained Language Models Understand Chinese Humor? In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pages 465– 480. ACM. find
  7. Shiyang Cheng, Pengcheng Zhu, Jueting Liu, and Zehua Wang. 2024. A Survey of Grapheme-to-Phoneme Conversion Methods. Applied Sciences, 14(24):11790. find
  8. Geoffrey Cideron, Sertan Girgin, Mauro Verzetti, Damien Vincent, Matej Kastelic, Zalán Borsos, Brian McWilliams, Victor Ungureanu, Olivier Bachem, Olivier Pietquin, Matthieu Geist, Léonard Hussenot, Neil Zeghidour, and Andrea Agostinelli. 2024. Mu-sicrl: Aligning Music Generation to Human Preference… find
  9. CMU. Carnegie mellon university pronouncing dictionary [online]. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, R. Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, and 1 others. 2025. Deepseek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv.org, a…
  10. DeepSeek-AI, A. Liu, Bei Feng, Bing Xue, Bing-Li Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, C. Deng, Chenyu Zhang, and 1 others. 2024. Deepseek-V3 Technical Report. arXiv.org, abs/2412.19437. find
  11. Avia Efrat, Or Honovich, and Omer Levy. 2023. LMentry: A language model benchmark of elementary language tasks. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10476– 10501, Toronto, Canada. Association for Computational Linguistics. find
  12. Anjie Fang, Simone Filice, Nut Limsopatham, and Oleg Rokhlenko. 2020. Using Phoneme Representations to Build Predictive Models Robust to ASR Errors. find
  13. In Proceedings of the 43rd International ACM SI-GIR Conference on Research and Development in Information Retrieval, pages 699–708. ACM. find
  14. J. R. Firch. 1957. Studies in Linguistic Analysis, volume 1. Wiley-Blackwell. find
  15. Tairan Fu, Raquel Ferrando, Javier Conde, Carlos Arriaga, and Pedro Reviriego. 2024. Why Do Large Language Models (LLMs) Struggle to Count Letters? arXiv.org, abs/2412.18626. — [arXiv:2412.18626](https://arxiv.org/abs/2412.18626) find
  16. Pablo Gervás. 2001. An expert system for the composition of formal Spanish poetry. Knowledge-Based Systems, 14(3-4):181–188. find
  17. Ariel Goldstein and Gabriel Stanovsky. 2024. Do zombies understand? a choose-your-own-adventure exploration of machine cognition. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7137–7143, Bangkok, Thailand. Association for Computational Linguistics. find
  18. Kyle Gorman, Lucas F.E. Ashby, Aaron Goyzueta, Arya McCarthy, Shijie Wu, and Daniel You. 2020. find
  19. The SIGMORPHON 2020 shared task on multilingual grapheme-to-phoneme conversion. In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 40–50, Online. Association for Computational Linguistics. find
  20. Ruiqi He, Yushu He, Longju Bai, Jiarui Liu, Zhenjie Sun, Zenghao Tang, He Wang, Hanchen Xia, Rada Mihalcea, and Naihao Deng. 2025. Chumor 2.0: Towards better benchmarking Chinese humor understanding from (ruo zhi ba). In Findings of the Association for Computational Linguistics: ACL 2025, pages 2179… find
  21. OpenAI Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o System Card. arXiv.org, abs/2410.21276.
  22. Jaehyung Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park, Sang-Woo Lee, Minjoon Seo, Jung-Woo Ha, and Jinwoo Shin. 2024. Sure: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs. In International Conference on Learning Representations (ICLR). find
  23. SungHo Kim, Nayeon Kim, Taehee Jeon, and SangKeun Lee. 2025. Polishing every facet of the GEM: Testing linguistic competence of LLMs and humans in Korean. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9955–9984, Vienna, Aust… find
  24. Yanis Labrak, Markus Frohmann, Gabriel Meseguer-Brocal, and Elena V. Epure. 2025. Synthetic lyrics detection across languages and genres. In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025), pages 524–541, Albuquerque, New Mexico. Association for Computational Linguistics. find
  25. Davor Lauc. 2025. Polyipa - Multilingual Phoneme-to-Grapheme Conversion Model. In Natural Language Processing, Information Retrieval and AI Trends 2025, volume abs/2412.09102, pages 143– 156. Academy & Industry Research Collaboration Center. — [arXiv:2412.09102](https://arxiv.org/abs/2412.09102) find
  26. Jackson L. Lee, Lucas F. E. Ashby, M. Elizabeth Garza, Yeonju Lee-Sikka, Sean Miller, Alan Wong, Arya D. McCarthy, and Kyle Gorman. 2020. Massively Multilingual Pronunciation Modeling with WikiPron. In International Conference on Language Resources and Evaluation (LREC). find
  27. Shunwei Lei, Yaoxun Xu, Zhiwei Lin, Huaicheng Zhang, Wei Tan, Hangting Chen, Jianwei Yu, Yixuan Zhang, Chenyu Yang, Haina Zhu, Shuai Wang, Zhiyong Wu, and Dong Yu. 2025. Levo: High-Quality Song Generation with Multi-Preference Alignment. arXiv.org, abs/2506.07520. — [arXiv:2506.07520](https://arxiv.org/abs/2506.07520) find
  28. Jiahao Li, Quan Wang, Zhendong Mao, Junbo Guo, Yanyan Yang, and Yongdong Zhang. 2022. Improving Chinese spelling check by character pronunciation prediction: The effects of adaptivity and granularity. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 42… find
  29. Zihong Liang, Xiaojun Quan, and Qifan Wang. 2023. Disentangled phonetic representation for Chinese spelling correction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13509– 13521, Toronto, Canada. Association for Computation… find
  30. Tyler Loakman, William Thorne, and Chenghua Lin. 2025. Comparing Apples to Oranges: A Dataset & Analysis of LLM Humour Understanding from Traditional Puns to Topical Jokes. arXiv. find
  31. Junyu Lu, Bo Xu, Xiaokun Zhang, Changrong Min, Liang Yang, and Hongfei Lin. 2023. Facilitating fine-grained detection of Chinese toxic language: Hierarchical taxonomy, resources, and benchmarks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… find
  32. Jianfei Ma, Zhaoxin Feng, Huacheng Song, Emmanuele Chersoni, and Zheng Chen. 2025. Reasoning or memorization? investigating LLMs’ capability in restoring Chinese Internet homophones. In Proceedings of the 3rd Workshop on Towards Knowledgeable Foundation Models (KnowFM), pages 120–139, Vienna, Austri… find
  33. Meta AI. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models [online]. 2024. find
  34. Ministry of Education and State Language Commission. 2013. List of Commonly Used Standard Chinese Characters. The Commercial Press. find
  35. State Language Commission. 2008. List of Frequently Used Words in Modern Chinese, volume 1. The Commercial Press. find
  36. Luca Moroni, Javier Aula-Blasco, Simone Conia, Irene Baucells, Naiara Perez, Silvia Paniagua Suárez, Anna Sallés, Malte Ostendorff, Júlia Falcão, Guijin Son, Aitor Gonzalez-Agirre, Roberto Navigli, and Marta Villegas. 2025. Multi-LMentry: Can multilingual LLMs solve elementary tasks across languages… find
  37. Yuchen Su, Yonghua Zhu, Ruofan Wang, Zijian Huang, Diana Benavides-Prado, and Michael Witbrock. 2025. A Survey of Pun Generation: Datasets, Evaluations and Methodologies. arXiv.org, abs/2507.04793. find
  38. Guanyi Mou, Pengyi Ye, and Kyumin Lee. 2020. Swe2: Subword Enriched and Significant Word Emphasized Framework for Hate Speech Detection. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1145– 1154. ACM. find
  39. Ashima Suvarna, Harshita Khandelwal, and Nanyun Peng. 2024. PhonologyBench: Evaluating phonological skills of large language models. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 1–14, Bangkok, Thailand. Association for Computational Linguistics. find
  40. Ziqian Ning, Huakang Chen, Yuepeng Jiang, Chunbo Hao, Guobin Ma, Shuai Wang, Jixun Yao, and Lei Xie. 2025. Diffrhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion. arXiv. open-dict data. ipa-dict - monolingual wordlists with pronunciation inf… find
  41. OpenAI. 2025. Openai o3 and o4-mini System Card. find
  42. Aitor Ormazabal, Mikel Artetxe, Manex Agirrezabal, Aitor Soroa, and Eneko Agirre. 2022. PoeLM: A meter- and rhyme-controllable language model for unsupervised poetry generation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3655–3670, Abu Dhabi, United Arab Emirates… find
  43. Kyubyong Park and Seanie Lee. 2020. g2pm: A Neural Grapheme-to-Phoneme Conversion Package for Mandarin Chinese Based on a New Open Benchmark Dataset. In Conference of the International Speech Communication Association (INTER-SPEECH), pages 1723–1727. find
  44. Brian Porter and Edouard Machery. 2024. Ai-generated poetry is indistinguishable from human-written poetry and is rated more favorably. Scientific Reports, 14(1). find
  45. Qwen Team. Qwq-32b: Embracing the power of reinforcement learning [online]. 2025. find
  46. Yesian Rohn. 2024. Duanzai: Slang-Enhanced LLM with Prompt for Humor Understanding. arXiv.org, abs/2405.15818. — [arXiv:2405.15818](https://arxiv.org/abs/2405.15818) find
  47. Hyopil Shin, Sangah Lee, Dongjun Jang, Wooseok Song, Jaeyoon Kim, Chaeyoung Oh, Hyemi Jo, Youngchae Ahn, Sihyun Oh, Hyohyeong Chang, and 1 others. 2025. Kobalt: Korean Benchmark For Advanced Linguistic Tasks. arXiv.org, abs/2505.16125. — [arXiv:2505.16125](https://arxiv.org/abs/2505.16125) find
  48. Yusen Sun, Liangyou Li, Qun Liu, and Dit-Yan Yeung. 2023. SongRewriter: A Chinese song rewriting system with controllable content and rhyme scheme. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12863–12880, Toronto, Canada. Association for Computational Linguistics. find
  49. Melanie Walsh, Anna Preus, and Maria Antoniak. 2024. Sonnet or not, bot? poetry evaluation for large models and datasets. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15568– 15603, Miami, Florida, USA. Association for Computational Linguistics. find
  50. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Conference on Neural Information Processing Systems (NeurIPS).
  51. Jörg Wöckener, Thomas Haider, Tristan Miller, The-Khang Nguyen, Thanh Tung Linh Nguyen, Minh Vu Pham, Jonas Belouadi, and Steffen Eger. 2021. End-to-end style-conditioned poetry generation: What does it take to learn from examples alone? In Proceedings of the 5th Joint SIGHUM Workshop on Computation… find
  52. Xiaofeng Wu, Karl Stratos, and Wei Xu. 2025. The impact of visual information in Chinese characters: Evaluating large models’ ability to recognize and utilize radicals. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Huma… find
  53. Yunze Xiao, Yujia Hu, Kenny Tsu Wei Choo, and Roy Ka-Wei Lee. 2024. ToxiCloakCN: Evaluating robustness of offensive language detection in Chinese with cloaking perturbations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6012–6025, Miami, Florida, U… find
  54. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, and 1 others. 2025a. Qwen2.5-Omni Technical Report. arXiv.org, abs/2503.20215.
  55. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xianzhong Shi, Ting He, Xinfa Zhu, and 1 others. 2025b. Qwen3-Omni Technical Report. arXiv.
  56. Nan Xu and Xuezhe Ma. 2025. LLM the genius paradox: A linguistic and math expert’s struggle with simple word-based counting problems. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1:…
  57. Zhijun Xu, Siyu Yuan, Lingjie Chen, and Deqing Yang. 2024. “a good pun is its own reword”: Can large language models understand puns? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11766–11782, Miami, Florida, USA. Association for Computational Lingu… find
  58. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 Technical Report. arXiv.org, abs/2505.09388.
  59. Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, and 1 others. 2024. Qwen2.5 Technical Report. arXiv.org, abs/2412.15115. find
  60. Chengyue Yu, Lei Zang, Jiaotuan Wang, Chenyi Zhuang, and Jinjie Gu. 2024. CharPoet: A Chinese classical poetry generation system based on token-free LLM. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 315–325, Bangk… find
  61. Zhiwei Yu, Hongyu Zang, and Xiaojun Wan. 2020. Homophonic pun generation with lexically constrained rewriting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2870–2876, Online. Association for Computational Linguistics. find
  62. Yixiao Yuan, Yangchen Huang, Yu Ma, Xinjin Li, Zhenglin Li, Yiming Shi, and Huapeng Zhou. 2024. Rhyme-aware Chinese lyric generator based on GPT. Fourth International Conference on Advanced Algorithms and Neural Networks (AANN 2024), 13416:134162P – 134162P–6. find
  63. Team Glm Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, and 1 others. 2024. Chatglm: A Family of Large Language Models from GLM-130b to GLM-4 All Tools. arXiv.org, abs/2406.12793.
  64. Yichao Zhou, Jyun-Yu Jiang, Jieyu Zhao, Kai-Wei Chang, and Wei Wang. 2020. “the boating store had its best sail ever”: Pronunciation-attentive contextualized pun recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 813–822, Online. Associatio… find
  65. Jian Zhu, Cong Zhang, and David Jurgens. 2022. Byt5 model for massively multilingual grapheme-to-phoneme conversion. In Conference of the International Speech Communication Association (IN-TERSPEECH), pages 446–450. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yue-2026-phun-bench-evaluating-llms-phonological/