Elamur Research⌕
☰
home / papers / 2026 / CORE: Measuring Multi-Agent LLM Interaction Quality under Game-Theoretic Pressures

CORE: Measuring Multi-Agent LLM Interaction Quality under Game-Theoretic Pressures

Paper · 2026 · Punya Syon Pandey, Yongjin Yang, Jiarui Liu et al.

Summary

Studies of LLM agents in game-theoretic settings usually focus on strategy or task success. They rarely measure how varied or degenerate the agents' language becomes. This paper proposes CORE (Conversational Robustness Evaluation), a single score in [0, 1] for the linguistic health of multi-agent dialogs that needs no task rewards or annotations. CORE multiplies three components:

  • Mode entropy: the normalized entropy of the distribution over conversational clusters, which detects mode collapse.
  • Repetition penalty: based on the share of repeated n-grams.
  • Semantic stagnation penalty: based on the cosine similarity of consecutive utterance embeddings (e.g., Sentence-BERT).

The two penalties are scaled by exponents set to the corpus's empirical Zipf and Heaps values. The authors ran pairwise dialogs between open-source LLMs under cooperative, competitive and neutral prompts. They analyzed the results with CORE alongside Zipf's and Heaps' Laws.

Cooperation produced both more repetition and faster vocabulary growth. Competition reduced repetition but kept vocabularies narrow. Neutral dialogs scored highest on CORE. The work offers a cheap diagnostic for mode collapse in agent debate and coordination systems.

Key points

  • CORE is the product of normalized mode entropy, an n-gram repetition penalty and a semantic stagnation penalty. The penalty terms are weighted by α (Zipf exponent) and β (Heaps exponent) estimated from the generated dialogs.
  • The tested models include LLaMa 3.1-8B Instruct, Qwen-3-8B, Qwen-2.5-7B Instruct, Mistral-7B v0.3, Mistral-7B Instruct, gemma-7b and gemma-7b-it.
  • Cooperative settings show steeper Zipf distributions and higher Heaps exponents. Competitive settings show lower values for both.
  • Neutral dialogs consistently reach the highest CORE. In turn-by-turn analysis, Mistral-7B-Instruct and Qwen2.5-7B-Instruct gave the richest exchanges. Two example dialogs illustrate the range: a collapsed, repetitive one scored about 0.002, and a diverse one about 0.45.
  • The study is limited to two-agent, short dialogs and used about 300 GPU hours on a single A100. Code is available at github.com/psyonp/core.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓University of TorontoToronto · Canada96 works · 142,007 citations🔬Vector InstituteToronto · Canada33 works · 4,535 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Punya Syon Pandey, Yongjin Yang, Jiarui Liu and Zhijing Jin. 2026. CORE: Measuring Multi-Agent LLM Interaction Quality under Game-Theoretic Pressures. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.57

@inproceedings{pandey2026core,
  title = {{CORE: Measuring Multi-Agent LLM Interaction Quality under Game-Theoretic Pressures}},
  author = {Punya Syon Pandey, Yongjin Yang, Jiarui Liu and Zhijing Jin},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.57},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.57}
}

References

31 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wenyue Hua, Ollie Liu, Lingyao Li, Alfonso Amayuelas, Julie Chen, Lucas Jiang, Mingyu Jin, Lizhou Fan, Fei Sun, William Wang, Xintong Wang, and Yongfeng Zhang. 2024. Game-theoretic llm: Agent workflow for negotiation games. find
  2. Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. 2025. Playing repeated games with large language models. Nature Human Behaviour, 9(7):1380–1390. find
  3. Ramon Ferrer i Cancho and Ricard V. Solé. 2003. Least effort and the origins of scaling in human language. Proceedings of the National Academy of Sciences, 100(3):788–791. find
  4. Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A. Ortega, DJ Strouse, Joel Z. Leibo, and Nando de Freitas. 2019. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. find
  5. Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh G… find
  6. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  7. Yipeng Kang, Tonghan Wang, and Gerard de Melo. 2020. Incorporating pragmatic reasoning communication into emergent language. In Advances in Neural Information Processing Systems, volume 33, pages 10348–10359. Curran Associates, Inc. find
  8. Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. 2017. Multi-agent cooperation and the emergence of (natural) language. find
  9. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17…
  10. Shaoguang Mao, Yuzhe Cai, Yan Xia, Wenshan Wu, Xun Wang, Fengyi Wang, Tao Ge, and Furu Wei. 2024. Alympics: Llm agents meet game theory – exploring strategic decision-making with ai agents.
  11. Patrick E. McKnight and Julius Najab. 2010. Mann-Whitney U Test, pages 1–1. John Wiley and Sons, Ltd. find
  12. Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie… find
  13. Meta. 2024b. Llama 3.1 8B Instruct. https://huggingface.co/meta-llama/Llama-3. 1-8B-Instruct. Accessed: July 2025. ↗
  14. Martin A. Nowak, Natalia L. Komarova, and Partha Niyogi. 2001. Evolution of universal grammar. Science, 291(5501):114–118. find
  15. Steven T. Piantadosi. 2014. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic Bulletin & Review, 21(5):1112–1130. find
  16. Giorgio Piatti, Zhijing Jin, Max Kleiman-Weiner, Bernhard Schölkopf, Mrinmaya Sachan, and Rada Mihalcea. 2024. Cooperate or collapse: Emergence of sustainable cooperation in a society of llm agents. find
  17. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… find
  18. Shannon Zejiang Shen, Hunter Lang, Bailin Wang, Yoon Kim, and David Sontag. 2024. Learning to decode collaboratively with multiple language models. find
  19. Eric Alden Smith. 2010. Communication and collective action: language and the evolution of human cooperation. Evolution and Human Behavior, 31(4):231– 245. find
  20. Jiakai Tang, Heyang Gao, Xuchen Pan, Lei Wang, Haoran Tan, Dawei Gao, Yushuo Chen, Xu Chen, Yankai Lin, Yaliang Li, Bolin Ding, Jingren Zhou, Jun Wang, and Ji-Rong Wen. 2025. Gensim: A general social simulation platform with large language model based agents. find
  21. Cooperative: Agents work together efficiently to solve a puzzle. find
  22. Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O’Sullivan, and Hoang D. Nguyen. 2025. Multi-agent collaboration mechanisms: A survey of llms. find
  23. Competitive: Agents negotiate to outperform their opponent. find
  24. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579–2605. find
  25. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… — Qwen3 Technical Report
  26. Neutral: Agents engage in open-ended, casual conversation. For each condition, dialogs of fixed length (10 turns) are generated. Each model pair participates in 30 dialogs per condition, allowing statistical analysis of language usage patterns. A.2 We selected eight distinct pretrained causal langua… find
  27. Yiqun Zhang, Xiaocui Yang, Shi Feng, Daling Wang, Yifei Zhang, and Kaisong Song. 2024a. Can llms beat humans in debating? a dynamic multi-agent framework for competitive debate. find
  28. Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Ö. Arik. 2024b. Chain of agents: Large language models collaborating on long-context tasks. find
  29. Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie. 2024. Competeai: Understanding the competition dynamics in large language model-based agents. find
  30. Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji, and Jiaxuan You. 2025. Multiagentbench: Evaluating the collaboration and competition of llm agents. find
  31. Qwen (3-8B and 2.5-7B instruction-tuned) • Mistral (7B and instruction-tuned) Models and their tokenizers are loaded on available hardware (GPU if available, otherwise CPU) using Hugging Face Transformers. Models are converted to half precision (float16) for efficient inference. A.3 1. The current d… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pandey-2026-core-measuring-multi-agent-llm/