Elamur Research⌕
☰
home / papers / 2023 / Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs

Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs

Paper · 2023 · Andries Smit, Paul Duckworth, Nathan Grinsztajn et al.

Summary

This paper asks whether multi-agent debate (MAD) is actually worth its extra cost for multiple-choice question answering with LLMs. The authors built a unified, open-source framework and used it to compare several debate protocols against single-agent and ensembling baselines. The debate protocols were Society of Minds, Multi-Persona and ChatEval. The baselines were Self-consistency, Ensemble Refinement, Solo Performance Prompting and a reduced Medprompt without kNN exemplar retrieval. Each protocol was combined with different agent-level prompts, such as zero-shot, CoT, few-shot and FS-CoT. The benchmarks were medical (MedQA, PubMedQA, clinical MMLU) and reasoning-oriented (e.g. CosmosQA, CIAR). The main backbone was GPT-3.5-turbo, with additional checks on GPT-4 and Mixtral 8x7B. The authors measured accuracy, time and cost. In their original configurations, debate methods did not consistently beat cheaper ensembling such as Self-consistency or Medprompt, and they needed more API calls. The authors argue the gap reflects sensitivity to hyperparameters rather than a weakness inherent to debate. Adjusting how strongly agents are told to agree with each other made Multi-Persona the top system. This matters for anyone choosing an inference-time strategy under a fixed budget.

Key points

  • Medical datasets: MedQA (1,273 USMLE questions, 4-option version), PubMedQA (500 questions) and the clinical subset of MMLU (123 questions).
  • Reasoning datasets: CosmosQA (500 questions sampled from about 7,000) and CIAR (50 counter-intuitive arithmetic questions).
  • New technique, "agreement intensity": a prompt-level control over how much agents defer to one another. Tuning it moved Multi-Persona from the worst protocol on MedQA to the best, ahead of Medprompt and Self-consistency.
  • Transfer across models: hyperparameter settings tuned on GPT-3.5 carried over well to GPT-4 but not to Mixtral 8x7B. The authors suggest architectural differences as a possible cause.
  • Code: all protocols, configurations and evaluation scripts are released as the DebateLLM repository (github.com/instadeepai/DebateLLM, from InstaDeep). The paper appeared at ICML 2024.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

Cite

Cite as Andries Smit, Paul Duckworth, Nathan Grinsztajn, Thomas D. Barrett and Arnu Pretorius. 2023. Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs. arXiv:2311.17371.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.17371 · arXiv 2311.17371

@misc{smit2023should,
  title = {{Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs}},
  author = {Andries Smit, Paul Duckworth, Nathan Grinsztajn, Thomas D. Barrett and Arnu Pretorius},
  year = {2023},
  doi = {10.48550/arxiv.2311.17371},
  eprint = {2311.17371},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.17371}
}

References

34 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kumar, H., Musabirov, I., Reza, M., Shi, J., Wang, X., Williams, J. J., Kuzminykh, A., and Liut, M. Impact of guidance and interaction strategies for llm use on learner performance and perception, 2024. find
  2. Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  3. Lai, J., Gan, W., Wu, J., Qi, Z., and Yu, P. S. Large language models in law: A survey, 2023. bench Team, B. Chess state tracking, 2021. URL https://github.com/google/BIG-bench/ tree/main/bigbench/benchmark_tasks/chess_ state_tracking. Accessed: 2024-05-27. ↗
  4. Li, G., Hammoud, H. A. A. K., Itani, H., Khizbullin, D., and Ghanem, B. Camel: Communicative agents for ”mind” exploration of large scale language model society. arXiv preprint arXiv:2303.17760, 2023.
  5. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  6. Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Tu, Z., and Shi, S. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118, 2023.
  7. Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201, 2023.
  8. Liévin, V., Hother, C. E., and Winther, O. Can large language models reason about medical questions? arXiv preprint arXiv:2207.08143, 2022. find
  9. Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023.
  10. Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., Luo, R., McKinney, S. M., Ness, R. O., Poon, H., Qin, T., Usuyama, N., White, C., and Horvitz, E. Can generalist foundation models outcompete special-purpose tuning? case study in medicine, 202… find
  11. Fu, Y., Peng, H., Khot, T., and Lapata, M. Improving language model negotiation with self-play and in-context learning from ai feedback. arXiv preprint arXiv:2305.10142, 2023.
  12. Han, T., Adams, L. C., Papaioannou, J.-M., Grundmann, P., Oberhauser, T., Löser, A., Truhn, D., and Bressem, K. K. Medalpaca– an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247, 2023.
  13. OpenAI. GPT-4 Technical Report. Technical report, OpenAI, 2023.
  14. OpenAI et al. Gpt-4 technical report, 2024.
  15. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International conference on learning representations, 2021.
  16. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  17. Huang, L., Bras, R. L., Bhagavatula, C., and Choi, Y. Cosmos QA: machine reading comprehension with contextual commonsense reasoning. CoRR, abs/1909.00277, 2019. URL http://arxiv.org/abs/1909.00277. ↗
  18. Pardos, Z. A. and Bhandari, S. Learning gain differences between chatgpt and human tutor generated algebra hints. arXiv preprint arXiv:2302.06871, 2023.
  19. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., La… find
  20. Park, J. S., O’Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. find
  21. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark, 2023.
  22. Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., and Szolovits, P. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
  23. Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138, 2022.
  24. Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., and Lu, X. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146, 2019.
  25. Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617, 2023.
  26. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  27. Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022.
  28. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022.
  29. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  30. Wang, G., Yang, G., Du, Z., Fan, L., and Li, X. Clinicalgpt: Large language models finetuned with diverse medical data and comprehensive evaluation. arXiv preprint arXiv:2306.09968, 2023a.
  31. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. In International conference on learning representations, 2023b.
  32. Wang, Z., Mao, S., Wu, W., Ge, T., Wei, F., and Ji, H. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint arXiv:2307.05300, 2023c.
  33. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  34. Zhang, Y., Zhang, F., Yang, Z., and Wang, Z. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. arXiv preprint arXiv:2305.19420, 2023. — [arXiv:2305.19420](https://arxiv.org/abs/2305.19420) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/smit-2023-should-going-mad-look/