Elamur Research⌕
☰
home / papers / 2024 / MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models

MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models

Summary

Most LLM benchmarks test knowledge and logical reasoning. They rarely check whether a model grasps what a speaker means beyond the literal words, and the few pragmatic evaluations that exist are mostly English-only. MultiPragEval fills this gap with a pragmatic test suite in English, German, Korean, and Chinese.

The suite is grounded in Grice's Cooperative Principle. Its items are grouped under the maxims of quantity, quality, relation, and manner, plus a control category that checks purely literal comprehension. Each unit gives a short situational context and an utterance, followed by a five-option multiple-choice question about the intended meaning. Linguists created the items and adapted them across languages with culturally localized details; for example, Florida oranges become Jeju tangerines in the Korean version.

The authors evaluated 15 proprietary and open-source LLMs. Claude3-Opus ranked first in every language, and Solar-10.7B and Qwen1.5-14B were the strongest open models. The benchmark matters because it enables cross-lingual comparison of implicature understanding and reveals language-dependent gaps within individual models.

Key points

  • The suite has 300 question units per language, 1,200 in total, across five categories (four Gricean maxims plus literal meaning). It is released on GitHub (DojunPark/MultiPragEval).
  • Claude3-Opus beat GPT-4 by roughly 6–10 points in all four languages.
  • Solar-10.7B and Qwen1.5-14B matched or exceeded smaller proprietary models such as GPT-3.5, Claude3-Haiku, and Mistral-Small.
  • Kendall's Tau rankings showed high agreement with other benchmarks, strongest with MMLU and, unexpectedly, MATH.
  • Limitations:
    • There is no human baseline.
    • Coverage is restricted to implicature, so speech acts, presupposition, and politeness are not tested.
    • A single sampling temperature (0.5) was used for all models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Implicature

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.genbench-1.7 · arXiv 2406.07736

@inproceedings{park2024multiprageval,
  title = {{MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models}},
  author = {Park, Dojun and Lee, Jiwoo and Park, Seohyun and Jeong, Hyeyun and Koo, Youngeun and Hwang, Soonha and Park, Seonwoo and Lee, Sungeun},
  booktitle = {Proceedings of the 2nd GenBench Workshop on Generalisation (Benchmarking) in NLP},
  year = {2024},
  doi = {10.18653/v1/2024.genbench-1.7},
  eprint = {2406.07736},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.genbench-1.7}
}

References

39 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. find our card
  2. AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card. find
  3. Giorgio Arcara and Valentina Bambini. 2016. A test for the assessment of pragmatic abilities and cognitive substrates (apacs): Normative data and psychometric properties. Frontiers in psychology, 7:172889. find
  4. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609. find our card
  5. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. find
  6. Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023. Open llm leaderboard. https://huggingface.co/ spaces/HuggingFaceH4/open_llm_leaderboard. ↗
  7. Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. 2023. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310.19736. find our card
  8. Ljubisa Bojic, Predrag Kovacevic, and Milan Cabarkapa. 2023. Gpt-4 surpassing human performance in linguistic pragmatics. arXiv preprint arXiv:2312.09545. find
  9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. find our card
  10. Rishi Bommasani, Percy Liang, and Tony Lee. 2023. Holistic evaluation of language models. Annals of the New York Academy of Sciences, 1525(1):140–146. find
  11. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. find our card
  13. Ludger Hoffmann. 2010. Reader. de Gruyter. find
  14. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1–45. find our card
  15. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. find our card
  16. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anasta-sios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating llms by human preference. find
  17. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. find
  18. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. find our card
  19. Chiara Barattieri di San Pietro, Federico Frau, Veronica Mangiaterra, and Valentina Bambini. 2023. The pragmatic profile of chatgpt: assessing the pragmatic skills of a conversational agent. find
  20. Arpad E Elo and Sam Sloan. 1978. The rating of chess-players: Past and present. find
  21. Sprachwissenschaft: ein Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference o… find our card
  22. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. find
  23. Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, et al. 2023. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312.15166. find
  24. Robert M Krauss and Susan R Fussell. 1996. Social psychological models of interpersonal communication. Social psychology: Handbook of basic principles, pages 655–701. find
  25. Sang Kwon, Gagan Bhatia, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Beyond English: Evaluating LLMs for Arabic grammatical error correction. In Proceedings of ArabicNLP 2023, pages 101–119, Singapore (Hybrid). Association for Computational Linguistics. find
  26. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212. find our card
  27. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Annual Meeting of the Association for Computational Linguistics. find our card
  28. Dojun Park, Jiwoo Lee, Hyeyun Jeong, Seohyun Park, and Sungeun Lee. 2024. Pragmatic competence evaluation of large language models for korean. arXiv preprint arXiv:2403.12675. find our card
  29. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67. find
  30. Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungju Kim, et al. 2024. Hyperclova x technical report. arXiv preprint arXiv:2404.01954. find
  31. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. find
  32. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational… find our card
  33. Damien Sileo, Philippe Muller, Tim Van de Cruys, and Camille Pradel. 2022. A pragmatics-centered evaluation framework for natural language understanding. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2382–2394, Marseille, France. European Language Resources Ass… find our card
  34. Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheon-bok Park, Kang Min Yoo, and Stella Biderman. 2024. Kmmlu: Measuring massive multitask language understanding in korean. arXiv preprint arXiv:2402.11548. find
  35. Dan Sperber and Deirdre Wilson. 1986. Relevance: Communication and cognition, volume 142. Harvard University Press Cambridge, MA. find
  36. Settaluri Lakshmi Sravanthi, Meet Doshi, Tankala Pavan Kalyan, Rudra Murthy, Pushpak Bhattacharyya, and Raj Dabre. 2024. Pub: A pragmatics understanding benchmark for assessing llms’ pragmatics capabilities. arXiv preprint arXiv:2401.07078. find our card
  37. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. find
  38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. find
  39. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/park-2024-multiprageval-multilingual-pragmatic-evaluation-large/