Elamur Research⌕
☰
home / papers / 2026 / Simple Agents, Biased Judges: Efficient Multi-Party Dialogue Generation & The Evaluation Gap

Simple Agents, Biased Judges: Efficient Multi-Party Dialogue Generation & The Evaluation Gap

Paper · 2026 · Kunal Samanta, Faisal Tareque Shohan, Amine Trabelsi et al.

Summary

The paper examines whether LLM-as-a-Judge evaluators can stand in for human raters when assessing open-domain multi-party social conversation, a setting that has seen little study. The authors build MPOD, a deliberately lightweight generator. Rather than running a graph of separate agents, it keeps one shared rolling context. An LLM-based selector picks the next speaker, and that speaker's persona is swapped into the system prompt. A cosine-similarity filter (nomic-embed-text-v1) rejects candidate utterances that repeat what the same agent said recently. In human evaluation, MPOD running on open-weight models (Llama 3.3 70B, Gemma 3 27B) matches or beats the multi-agent ConvoGen baseline. When human preferences are compared with frontier LLM judges, agreement is barely above chance. Judges almost never declare ties, reward longer outputs, and favour assistant-style GPT text even when utterance order has been scrambled. The authors argue this undermines the common practice of validating synthetic dialogue data with general-purpose instruction-tuned judges. They call for evaluators trained on human social-preference data.

Key points

  • Across 319 pairwise comparisons, Cohen's κ between humans and LLM judges is about 0.11. Human–human agreement is higher (κ = 0.29), and LLM–LLM agreement is high (κ = 0.68). Judges are therefore consistent with one another but systematically diverge from humans.
  • In the Transplant Ablation, one GPT-5.1 agent is placed in an otherwise all-Llama 3.3-70B group. Gemini-3-Flash and GPT-5.2 preferred this mixed group over the pure Llama group on all 10 topics. A pure GPT-5.1 group in turn beat the mixed group on all 10.
  • In the shuffle stress test, GPT-5.1 conversations with randomized utterance order still won all 10 topics against Qwen 3 30B. This shows the judges are insensitive to coherence and turn structure.
  • Judge win rates rise with average utterance length (R² between 0.51 and 0.94), which points to strong verbosity bias.
  • MPOD setup:
    • The turn selector reads the last 5 utterances and outputs JSON, falling back to random selection in under 5% of turns.
    • The similarity filter uses window k = 5 and threshold τ = 0.5, allowing up to 5 resamples with temperature raised by 0.1 each time.
    • Open-weight models run with 4-bit weights.
    • The study is limited to English and three-participant conversations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université de SherbrookeSherbrooke · Canada4 works · 53 citations

Also on the byline: Université Laval.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 624 works

Cite

Cite as Kunal Samanta, Faisal Tareque Shohan, Amine Trabelsi and Richard Khoury. 2026. Simple Agents, Biased Judges: Efficient Multi-Party Dialogue Generation & The Evaluation Gap. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2006

@inproceedings{samanta2026simple,
  title = {{Simple Agents, Biased Judges: Efficient Multi-Party Dialogue Generation \& The Evaluation Gap}},
  author = {Kunal Samanta, Faisal Tareque Shohan, Amine Trabelsi and Richard Khoury},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2006},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2006}
}

References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shikib Mehri and Maxine Eskenazi. 2020. Unsupervised evaluation of interactive dialog with dialogpt. arXiv preprint arXiv:2006.12719.
  2. Jonas Becker. 2024. Multi-agent large language models for conversational task-solving. arXiv preprint arXiv:2410.22932. — [arXiv:2410.22932](https://arxiv.org/abs/2410.22932) find
  3. John Mendonça, Isabel Trancoso, and Alon Lavie. 2024. Soda-eval: Open-domain dialogue evaluation in the age of llms. arXiv preprint arXiv:2408.10902. find
  4. Zhi-Yuan Chen, Hao Wang, Xinyu Zhang, Enrui Hu, and Yankai Lin. 2025. Beyond the surface: Measuring self-preference in llm judgments. arXiv preprint arXiv:2506.02592.
  5. Cheng-Han Chiang and Hung-yi Lee. 2023. A closer look into automatic evaluation using large language models. arXiv preprint arXiv:2310.05657. — [arXiv:2310.05657](https://arxiv.org/abs/2310.05657) find
  6. Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6556–6576. find
  7. Gemini 3. 2025. A new era of intelligence with Gemini 3. https://blog.google/products/gemini/ gemini-3. Accessed: 2026-01-05. ↗
  8. Reem Gody, Mahmoud Goudy, and Ahmed Y Tawfik. 2025. Convogen: Enhancing conversational ai with synthetic data: A multi-agent approach. arXiv preprint arXiv:2503.17460. — [arXiv:2503.17460](https://arxiv.org/abs/2503.17460) find
  9. GPT 5.1. GPT-5.1 A smarter, more conversational ChatGPT. https://openai.com/index/ gpt-5-1/. Accessed: 2026-01-05. ↗
  10. GPT-OSS. Introducing gpt-oss. https://openai.com/index/introducing-gpt-oss/. Accessed: 2026-01-05. ↗
  11. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  12. Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Bras, Malihe Alikhani, Gunhee Kim, and 1 others. 2023. Soda: Million-scale dialogue distillation with social commonsense contextualization. In Proceedings of the 2023 Conference on Empirical Methods in Natural…
  13. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  14. Ryota Nonomura and Hiroki Mori. 2025. Who speaks next? multi-party ai discussion leveraging the systematics of turn-taking in murder mystery games. Frontiers in Artificial Intelligence, 8:1582287. find
  15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  16. Eric Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston. 2022. Human evaluation of conversations is an open problem: comparing the sensitivity of various methods for evaluating dialogue agents. In Proceedings of the 4th Workshop on NLP for Conversational AI, pages 77– 97… find
  17. Sathya Krishnan Suresh, Wu Mengjun, Tushar Pranav, and EngSiong Chng. 2025. Diasynth: Synthetic dialogue generation framework for low resource dialogue applications. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 673–690. find
  18. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  19. Hongru Wang, Lingzhi Wang, Yiming Du, Liang Chen, Jingyan Zhou, Yufei Wang, and Kam-Fai Wong. 2023. A survey of the evolution of language model-based dialogue systems. arXiv preprint arXiv:2311.16789. — [arXiv:2311.16789](https://arxiv.org/abs/2311.16789) find
  20. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, and 1 others. 2024. Autogen: Enabling next-gen llm applications via multi-agent conversations. In First Conference on Language Modeling. find
  21. Andrea Wynn, Harsh Satija, and Gillian Hadfield. 2025. Talk isn’t always cheap: Understanding failure modes in multi-agent debate. arXiv preprint arXiv:2509.05396. — [arXiv:2509.05396](https://arxiv.org/abs/2509.05396) find
  22. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  23. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/samanta-2026-simple-agents-biased-judges-efficient/