Elamur Research
☰
home / papers / 2023 / Improving Factuality and Reasoning in Language Models through Multiagent Debate

Improving Factuality and Reasoning in Language Models through Multiagent Debate

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2305.14325

Summary

Large language models often state invented facts with confidence or make unjustified leaps in their reasoning. Earlier remedies, such as chain-of-thought prompting, self-consistency and reflection, all work with a single model instance. Du et al. instead treat several copies of an LLM as agents. Each agent first answers a query independently. It then reads the other agents' answers, critiques them and revises its own, and this exchange repeats over several rounds until the group reaches a common answer.

The method needs only black-box text generation and uses identical prompts for every task. It can also be stacked with zero-shot chain of thought. On six benchmarks (arithmetic, grade-school math, MMLU, chess move validity and optimality, and factual biographies), debate outperformed single-model baselines, including zero-shot chain of thought and reflection. Increasing both the number of agents and the number of rounds helped. The authors also introduce a new benchmark of computer scientist biographies. The approach matters because it improves accuracy without access to model internals and could produce better data for model self-improvement.

Key points

  • Six evaluation tasks: Arithmetic, Grade School Math (GSM8K), MMLU, Chess Move Validity, Chess Move Optimality and Biographies. The Biographies benchmark is a new dataset on famous computer scientists, where models frequently get institutions and dates wrong.
  • Agents often begin with different answers but nearly always converge on one shared answer. In some cases every agent starts out wrong and the group still reaches the correct result. Facts the agents are unsure about tend to be disputed and then dropped.
  • In a mixed-model experiment, chatGPT and Bard debated 20 GSM8K problems. Bard alone solved 11, chatGPT alone solved 14, and their joint debate solved 17.
  • The method needs no likelihoods or gradients, so it works with public model APIs. It is also complementary to retrieval and prompt engineering.
  • Limitations:
    • Debate requires many more model calls, which makes it expensive.
    • In long debates, models tend to attend mainly to the most recent messages.
    • Converged answers can still be wrong, and models then wrongly insist they are correct.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

469 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 448 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.14325 · arXiv 2305.14325

@misc{du2023improving,
  title = {{Improving Factuality and Reasoning in Language Models through Multiagent Debate}},
  author = {Du, Yilun and Li, Shuang and Torralba, Antonio and Tenenbaum, Joshua B. and Mordatch, Igor},
  year = {2023},
  doi = {10.48550/arxiv.2305.14325},
  eprint = {2305.14325},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.14325}
}

References

33 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: A visual language model for few-shot learning. NeurIPS, 2022. URL https://arxiv.org/abs/2204.14198. 9 ↗ our card
  2. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Neural Information Processing Systems, 2017. 9 find
  3. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 5, 9 find our card
  4. Y. Du, S. Li, and I. Mordatch. Compositional visual generation with energy based models. In Advances in Neural Information Processing Systems, 2020. 9 find
  5. Y. Du, C. Durkan, R. Strudel, J. B. Tenenbaum, S. Dieleman, R. Fergus, J. Sohl-Dickstein, A. Doucet, and W. Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. arXiv preprint arXiv:2302.11552, 2023. 9 — [arXiv:2302.11552](https://arxiv.org/abs/2302.11552) find
  6. Fsmosca. Fsmosca/pgn-standard: Portable game notation specification and implementation guide. URL https://github.com/fsmosca/PGN-Standard. 5 ↗
  7. K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang. REALM: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. 9 find
  8. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. 7 find our card
  9. G. Irving, P. Christiano, and D. Amodei. Ai safety via debate. arXiv preprint arXiv:1805.00899, 2018. 9 find
  10. S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. H. Dodds, N. DasSarma, E. Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. 7, 9 find
  11. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022. 2, 6, 9 find our card
  12. N. Lee, W. Ping, P. Xu, M. Patwary, P. N. Fung, M. Shoeybi, and B. Catanzaro. Factuality enhanced language models for open-ended text generation. Advances in Neural Information Processing Systems, 35:34586–34599, 2022. 9 find
  13. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022. 9 find our card
  14. S. Li, Y. Du, J. B. Tenenbaum, A. Torralba, and I. Mordatch. Composing ensembles of pre-trained models via iterative consensus. arXiv preprint arXiv:2210.11522, 2022. 9 — [arXiv:2210.11522](https://arxiv.org/abs/2210.11522) find
  15. Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode. Science, 378 (6624):1092–1097, 2022. 5, 9 find our card
  16. H. Liu, L. Lee, K. Lee, and P. Abbeel. Instruction-following agents with jointly pre-trained vision-language models. arXiv preprint arXiv:2210.13431, 2022. 9 — [arXiv:2210.13431](https://arxiv.org/abs/2210.13431) find
  17. N. Liu, S. Li, Y. Du, A. Torralba, and J. B. Tenenbaum. Compositional visual generation with composable diffusion models. arXiv preprint arXiv:2206.01714, 2022. 9 — [arXiv:2206.01714](https://arxiv.org/abs/2206.01714) find
  18. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. 2, 5, 9 find
  19. M. Minsky. Society of mind. Simon and Schuster, 1988. 1 find
  20. M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. 9 find our card
  21. OpenAI. Chatgpt: Optimizing language models for dialogue, Dec 2022. URL https:// openai.com/blog/chatgpt/. 2, 5 find
  22. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. 4 find
  23. S. Pichai. An important next step on our ai journey, Feb 2023. URL https://blog.google/ technology/ai/bard-google-ai-search-updates/. 2, 9 ↗
  24. N. F. Rajani, B. McCann, C. Xiong, and R. Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361, 2019. 9 find our card
  25. L. Reynolds and K. McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7, 2021. 9 find our card
  26. N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023. 2, 5, 9 find our card
  27. A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. 7, 14 find our card
  28. R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. 9 find our card
  29. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. 9 find our card
  30. J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. 9 find our card
  31. E. Zelikman, Y. Wu, J. Mu, and N. Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022. 9 find our card
  32. A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022. URL https://arxiv.org/abs/2204. 00598. 9 ↗ our card
  33. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 9 find