Elamur Research⌕
☰
home / papers / 2025 / Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoning

Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoning

Paper · 2025 · Sugyeong Eo, Hyeonseok Moon, Evelyn Hayoon Zi et al.

Summary

Multiagent debate, where several LLM agents exchange and revise answers, can sharpen reasoning. However, every extra round multiplies inference cost, and needless argument can let one agent's mistake spread to the others. The authors introduce DOWN (Debate Only When Necessary). In DOWN, a single agent answers first and the system measures how certain that agent is, using the mean token probability of its output or, for models without logit access, a verbalized confidence. If the score is above a tuned threshold, the first answer is kept. Otherwise, a two-round, three-agent debate begins. Each agent sees its peers' answers alongside their confidence scores, and the outcome is decided by majority vote or a judge agent. The authors test DOWN with roughly 8B and 70B open models and GPT-4o-mini, in both homogeneous and mixed-agent setups, on MUSR and StrategyQA, with MMLU used to check generalization. DOWN reduces agent calls as much as sixfold while matching or exceeding full-debate baselines. This makes collaborative reasoning cheaper and limits error propagation.

Key points

  • Four-stage pipeline: (1) initial answer with a confidence score, (2) threshold check, (3) confidence-guided debate (2 rounds × 3 agents), and (4) final answer by voting or a judge agent.
  • Efficiency: full Debate always needs about six agent calls per query. DOWN often stops after one call and still achieves comparable or higher accuracy.
  • Threshold tuning: shifting the threshold θ by ±0.1 shows a trade-off between accuracy and cost. The combined scores were 0.29 for θ*−0.1, 1.19 for θ*, and 1.00 for θ*+0.1.
  • Generalization: on MMLU with GPT-4o-mini, DOWN performs on par with Debate across six domains at much lower cost.
  • Error propagation: in a MUSR example, Debate and MAD start with a correct answer but revise it into a wrong one. DOWN skips debate because its confidence is 0.91 and keeps the correct answer. The authors note two limitations: experiments are English-only, and GPT-4o-mini was used instead of GPT-4o.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Korea UniversitySeoul · South Korea30 works · 9,494 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Sugyeong Eo, Hyeonseok Moon, Evelyn Hayoon Zi, Chanjun Park and Heuiseok Lim. 2025. Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoning. arXiv:2504.05047.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2504.05047 · arXiv 2504.05047

@misc{eo2025debate,
  title = {{Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoning}},
  author = {Sugyeong Eo, Hyeonseok Moon, Evelyn Hayoon Zi, Chanjun Park and Heuiseok Lim},
  year = {2025},
  doi = {10.48550/arxiv.2504.05047},
  eprint = {2504.05047},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.05047}
}

References

32 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2024. Chateval: Towards better LLM-based evaluators through multi-agent debate. In The Twelfth International Conference on Learning Representations.
  2. Justin Chen, Swarnadeep Saha, and Mohit Bansal. 2024a. ReConcile: Round-table conference improves reasoning via consensus among diverse LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7066–7085, Bangkok, Thailand. Associ…
  3. Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2024b. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelf…
  4. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages…
  5. Lizhou Fan, Lingyao Li, Zihui Ma, Sanggyu Lee, Huizi Yu, and Libby Hemphill. 2024. A bibliometric review of large language models research from 2017 to 2023. ACM Transactions on Intelligent Systems and Technology, 15(5):1–25. find
  6. Yi Fang, Moxin Li, Wenjie Wang, Lin Hui, and Fuli Feng. 2025. Counterfactual debating with preset stances for hallucination elimination of LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10554–10568, Abu Dhabi, UAE. Association for Computational Linguist… find
  7. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. find
  8. Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nitya Nadgir, and Arvind Narayanan. 2024. Ai agents that matter. arXiv preprint arXiv:2407.01502.
  9. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc.
  10. Huao Li, Yu Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Charles Lewis, and Katia Sycara. 2023. Theory of mind for multi-agent collaboration via large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 180–192, Singapore. Assoc…
  11. Renhao Li, Minghuan Tan, Derek F. Wong, and Min Yang. 2024a. CoEvol: Constructing better responses for instruction finetuning through multi-agent cooperation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4703–4721, Miami, Florida, USA. Association… find
  12. Wenzhe Li, Yong Lin, Mengzhou Xia, and Chi Jin. 2025. Rethinking mixture-of-agents: Is mixing different large language models beneficial? arXiv preprint arXiv:2502.00674. — [arXiv:2502.00674](https://arxiv.org/abs/2502.00674) find
  13. Yunxuan Li, Yibing Du, Jiageng Zhang, Le Hou, Peter Grabowski, Yeqing Li, and Eugene Ie. 2024b. Improving multi-agent debate with sparse communication topology. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7281–7294, Miami, Florida, USA. Association for Computation… find
  14. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17…
  15. Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. 2024. A dynamic LLM-powered agent network for task-oriented agent collaboration. In First Conference on Language Modeling.
  16. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… find
  17. Marvin Minsky. 1988. Society of mind. Simon and Schuster. find
  18. Ali Razghandi, Seyed Mohammad Hadi Hosseini, and Mahdieh Soleymani Baghshah. 2025. Cer: Confidence enhanced reasoning in llms. arXiv preprint arXiv:2502.14634.
  19. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314.
  20. Zayne Rea Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. 2024. MuSR: Testing the limits of chain-of-thought with multistep soft reasoning. In The Twelfth International Conference on Learning Representations.
  21. Haoyang Su, Renqi Chen, Shixiang Tang, Xinzhe Zheng, Jingzhe Li, Zhenfei Yin, Wanli Ouyang, and Nanqing Dong. 2024. Two heads are better than one: A multi-agent system has the potential to improve scientific idea generation. arXiv preprint arXiv:2410.09403. — [arXiv:2410.09403](https://arxiv.org/abs/2410.09403) find
  22. Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. 2025. Confidence improves self-consistency in llms. arXiv preprint arXiv:2502.06233. find
  23. Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O’Sullivan, and Hoang D Nguyen. 2025. Multi-agent collaboration mechanisms: A survey of llms. arXiv preprint arXiv:2501.06322.
  24. Junlin Wang, Jue WANG, Ben Athiwaratkun, Ce Zhang, and James Zou. 2025. Mixture-of-agents enhances large language model capabilities. In The Thirteenth International Conference on Learning Representations.
  25. Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. 2024. Rethinking the bounds of LLM reasoning: Are multi-agent discussions the key? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6106–6131, Bangkok, Thailand…
  26. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
  27. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155.
  28. Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. 2023. Examining inter-consistency of large language models collaboration: An in-depth analysis via debate. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7572–7590, Singapore. Association for Computational Ling…
  29. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.
  30. Zhangyue Yin, Qiushi Sun, Cheng Chang, Qipeng Guo, Junqi Dai, Xuanjing Huang, and Xipeng Qiu. 2023. Exchange-of-thought: Enhancing large language model capabilities through cross-model communication. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 151… find
  31. Wei Zhou, Mohsen Mesgar, Annemarie Friedrich, and Heike Adel. 2025. Efficient multi-agent collaboration with tool use for online planning in complex table question answering. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 945–968, Albuquerque, New Mexico. Association… find
  32. Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R. Ashley, Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, Louis Kirsch, Bing Li, Guohao Li, Shuming Liu, Jinjie Mai, Piotr Pi˛ekos, Aditya Ramesh, Imanol Schlag, Weimin Shi, Aleksan… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/eo-2025-debate-only-necessary-adaptive-multiagent/