Elamur Research⌕
☰
home / papers / 2024 / Pragmatic Competence Evaluation of Large Language Models for the Korean Language

Pragmatic Competence Evaluation of Large Language Models for the Korean Language

Summary

Most LLM benchmarks focus on knowledge, reasoning, and literal meaning. They also tend to be English-centric and rely heavily on multiple-choice formats. This paper examines a neglected skill: whether models recover what a speaker implies rather than what the words literally say, tested in Korean. The authors built a 120-item test set based on Grice's four conversational maxims (quantity, quality, relation, manner). Each item pairs a brief scene and an utterance with a question about the utterance's intended meaning. Models were assessed with automatically scored multiple-choice questions (MCQs) and with open-ended questions (OEQs) graded by human experts. The study also measured the effect of few-shot and Chain-of-Thought (CoT) prompting. GPT-4 scored highest in both formats, with HyperCLOVA X close behind and Gemini-Pro next. Format strongly influenced rankings. Few-shot examples generally helped, while CoT nudged models toward literal readings and hurt performance. The work provides the first dedicated public resource for Korean pragmatic evaluation. It also shows that culture-specific knowledge matters for inferring implied meaning.

Key points

  • The test set has 120 units, 30 for each Gricean maxim. Four experts with master's degrees in linguistics or related fields wrote the items, and the set is released on GitHub (DojunPark/pragmatic_eval_korean).
  • GPT-4 reached 81.11 on MCQs and 85.69 on OEQs. HyperCLOVA X narrowed its MCQ gap to GPT-4 in the open-ended setting.
  • The open-source LDCC-Solar beat GPT-3.5 on MCQs but fell behind it on OEQs, which shows how much question format shapes results.
  • In the error analysis, the most common failure in open answers was missing the implicature even when the context was understood. Purely literal readings came next, especially for the maxim of manner. Gemini-Pro sometimes printed raw markdown symbols in its answers.
  • On culture-bound items, such as cash-only bung-eo-bbang street stalls or tangerine gifts from Jeju Island, HyperCLOVA X often outperformed GPT-4. This agrees with earlier KMMLU findings on Korea-specific knowledge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Implicature

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2403.12675

@inproceedings{park2024pragmatic,
  title = {{Pragmatic Competence Evaluation of Large Language Models for the Korean Language}},
  author = {Park, Dojun and Lee, Jiwoo and Jeong, Hyeyun and Park, Seohyun and Lee, Sungeun},
  booktitle = {Proceedings of the 38th Pacific Asia Conference on Language, Information and Computation},
  year = {2024},
  eprint = {2403.12675},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.12675}
}

References

40 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. find our card
  2. Giorgio Arcara and Valentina Bambini. 2016. A test for the assessment of pragmatic abilities and cognitive substrates (apacs): Normative data and psychometric properties. Frontiers in psychology, 7:70. find
  3. Chiara Barattieri di San Pietro, Federico Frau, Veronica Mangiaterra, and Valentina Bambini. 2023. The pragmatic profile of chatgpt: assessing the pragmatic skills of a conversational agent. find
  4. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234. find our card
  5. Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. 2024. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_ leaderboard. ↗
  6. Yingqiang Ge, Wenyue Hua, Kai Mei, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang, et al. 2024. Openagi: When llm meets domain experts. Advances in Neural Information Processing Systems, 36. find
  7. Herbert Paul Grice. 1975. Logic and conversation. Syntax and semantics, 3:43–58. find
  8. Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. 2023. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310.19736. find our card
  9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. find our card
  10. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Sort, 2(4):0–6. find our card
  11. Ludger Hoffmann. 2010. Reader. de Gruyter. find
  12. Sprachwissenschaft: ein Ljubisa Bojic, Predrag Kovacevic, and Milan Cabarkapa. 2023. Gpt-4 surpassing human performance in linguistic pragmatics. arXiv preprint arXiv:2312.09545. find
  13. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.08745, 1(10). find our card
  14. Rishi Bommasani, Percy Liang, and Tony Lee. 2023. Holistic evaluation of language models. Annals of the New York Academy of Sciences, 1525(1):140–146. find
  15. Jad Kabbara. 2019. Computational investigations of pragmatic effects in natural language. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 71–76. find
  16. Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint ArXiv:2005.14165. find
  17. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1–45. find our card
  18. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. find
  19. Aisha Khatun and Daniel G Brown. 2024. A study on large language models’ limitations in multiple-choice question answering. arXiv preprint arXiv:2401.07955. — [arXiv:2401.07955](https://arxiv.org/abs/2401.07955) find
  20. Wonchul Kim. 2024. Ldcc-solar-10.7b. https://huggingface.co/LDCC/LDCC-SOLAR-10.7B. ↗
  21. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213. find our card
  22. Shalom Lappin. 2024. Assessing the strengths and weaknesses of large language models. Journal of Logic, Language and Information, 33(1):9–20. find
  23. James Manyika and Sissie Hsiao. 2023. An overview of bard: an early experiment with generative ai. AI. Google Static Documents, 2. find
  24. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Ab… find our card
  25. Yunju Nam, Hyenyeong Chung, and Upyong Hong. 2023. Language artificial intelligences’ communicative performance quantified through the gricean conversation theory. Cyberpsychology, Behavior, and Social Networking, 26(12):919–923. find
  26. Graziella Orrù, Andrea Piarulli, Ciro Conversano, and Angelo Gemignani. 2023. Human-like problem-solving abilities in large language models using chatgpt. Frontiers in artificial intelligence, 6:1199350. find
  27. Chanjun Park, Hyeonwoo Kim, Dahyun Kim, SeongH-wan Cho, Sanghoon Kim, Sukyung Lee, Yungi Kim, and Hwalsuk Lee. 2024. Open Ko-LLM leaderboard: Evaluating large language models in Korean with Koh5 benchmark. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vo… find
  28. R.S. Satpute and A. Agrawal. 2022. Pragmatic analysis in natural language processing. Journal of Basic Sciences, 22(12). find
  29. SM Seals and Valerie L Shalin. 2023a. Discourse over discourse: The need for an expanded pragmatic focus in conversational ai. arXiv preprint arXiv:2304.14543. — [arXiv:2304.14543](https://arxiv.org/abs/2304.14543) find
  30. SM Seals and Valerie L Shalin. 2023b. Expanding the set of pragmatic considerations in conversational ai. arXiv preprint arXiv:2310.18435. — [arXiv:2310.18435](https://arxiv.org/abs/2310.18435) find
  31. Muhammad Shidiq. 2023. The use of artificial intelligence-based chat-gpt and its challenges for the world of education; from the viewpoint of the development of creative writing skills. In Proceeding of international conference on education, society and humanity, volume 1, pages 353–357. find
  32. Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Carina Negreanu, and Gust Verbruggen. 2023. Code-Fusion: A pre-trained diffusion model for code generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11697–11708, Singapore. Association for Com… find
  33. Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheon-bok Park, Kang Min Yoo, and Stella Biderman. 2024. Kmmlu: Measuring massive multitask language understanding in korean. arXiv preprint arXiv:2402.11548. find
  34. Frans Sudirjo, Karno Diantoro, Jassim Ahmad Al-Gasawneh, Hizbul Khootimah Azzaakiyyah, and Abu Muna Almaududi Ausat. 2023. Application of chatgpt in improving customer sentiment analysis for businesses. Jurnal Teknologi Dan Sistem Informasi Bisnis, 5(3):283–288. find
  35. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. find our card
  36. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. find
  37. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837. find our card
  38. Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. 2024. Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data, 18(6):1–32. find our card
  39. Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungju Kim, et al. 2024. Hyperclova x technical report. arXiv preprint arXiv:2404.01954. find
  40. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/park-2024-pragmatic-competence-evaluation-large-language/