Elamur Research⌕
☰
home / papers / 2026 / Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling

Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling

Paper · 2026 · Florian Valentin Wunderlich, Lars Benedikt Kaesberg, Jan Philip Wahle et al.

Summary

Inference-time techniques can raise LLM accuracy without retraining, but they are usually reported by accuracy alone, which hides large differences in compute use. This study compares four test-time scaling strategies under matched budgets: self-consistency, self-refinement, multi-agent debate and mixture-of-agents (MoA). The authors vary the number of sampled sequences, refinement iterations, agents, rounds, proposer models and layers. They also vary model size (8B and 70B). Cost is measured as runtime per task, combining FLOPs with memory-transfer time rather than FLOPs alone. Evaluation covers MMLU-Pro and BBH, and Pareto fronts identify the cheapest configuration for each accuracy level.

Multi-agent pipelines proved the most compute-efficient overall, and MoA dominated the Pareto front. Additional inference compute paid off most on difficult questions. The paper turns these results into simple rules for deployments with limited compute:

  • For debate, add agents rather than rounds.
  • For MoA, use one more proposer model than layers.

Key points

  • The study ran 34 configurations and more than 100 evaluations on 1,000-item samples of MMLU-Pro and BBH. The authors estimate each data point's 95% confidence interval at about ±3 points.
  • On MMLU-Pro, the 70B model rose from 64.3% with CoT to 71.4% with MoA at about 20× the CoT budget, a gain of 7.1 points. At similar cost, self-consistency reached 68.7% and debate 70.0%. Roughly 5× the CoT budget already captures most of the MoA gain, and self-consistency remains the best option at or below about 3×.
  • Harder MMLU-Pro tasks gained about +8.5 to +9 points at 15–20× the CoT budget, versus +2.2 for easy ones. The authors argue this supports routing compute adaptively by task difficulty.
  • Using a larger model without test-time scaling can be more efficient than heavily scaling a smaller model.
  • In MoA, mixing 8B and 70B proposers is generally worse than keeping proposer sizes uniform:
    • Accuracy declines steadily as the share of 8B proposers grows.
    • A mixed MoA with three 70B and two 8B proposers scored 67.5%.
    • An all-70B MoA with three proposers and two layers scored 70.1%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GöttingenGöttingen · Germany20 works · 158 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksHardware and Architecture · 167 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Florian Valentin Wunderlich, Lars Benedikt Kaesberg, Jan Philip Wahle, Terry Ruas and Bela Gipp. 2026. Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.1

@inproceedings{wunderlich2026multi,
  title = {{Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling}},
  author = {Florian Valentin Wunderlich, Lars Benedikt Kaesberg, Jan Philip Wahle, Terry Ruas and Bela Gipp},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.1},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.1}
}

References

30 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Our compute costs were estimated using theoretical formulas that account for FLOPs and memory transfer. This provides a consistent basis for comparison; however, real-world runtime may vary due to framework overhead, batching strategies, or system-level constraints. These practical considerations al… find
  2. Jonas Becker, Lars Benedikt Kaesberg, Andreas Stephan, Jan Philip Wahle, Terry Ruas, and Bela Gipp. 2025b. Stay Focused: Problem Drift in Multi-Agent Debate. Preprint, arXiv:2502.19559. find
  3. Jonas Becker, Lars Benedikt Kaesberg, Niklas Bauer, Jan Philip Wahle, Terry Ruas, and Bela Gipp. 2025a. MALLM: Multi-Agent Large Language Models Framework. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 418–439, Suzhou, China.… find
  4. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2024. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs.
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. Preprint, arXiv:2110.14168.
  6. Dujian Ding, Ankur Mallick, Shaokun Zhang, Chi Wang, Daniel Madrigal, Mirian Del Carmen Hipolito Garcia, Menglin Xia, Laks V. S. Lakshmanan, Qingyun Wu, and Victor Rühle. 2025. BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute. find
  7. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. Preprint, arXiv:2305.14325.
  8. Finally, our study was limited to non-reasoning models. Future work could explore how reasoning models interact with inference-time scaling strategies, for example, whether their stronger reasoning abilities change the compute-performance trade-offs and what patterns (e.g., reasoning strength, pipel… find
  9. Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Yifu Lu, Mengdi Wang, Dinesh Manocha, Furong Huang, Mohammad Ghavamzadeh, and Amrit Singh Bedi. 2025. Does Thinking More always Help? Understanding Test-Time Scaling in Reasoning Models. find
  10. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large Language Models Cannot Self-Correct Reasoning Yet. Preprint, arXiv:2310.01798.
  11. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2022. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. Preprint, arXiv:2210.09261.
  12. Yunho Jin, Gu-Yeon Wei, and David Brooks. 2025. The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Compute. find
  13. J. Wahle, T. Ruas, S. M. Mohammad, N. Meuschke, and B. Gipp. 2023. Ai usage cards: Responsibly reporting ai-generated content. In 2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 282–284, Los Alamitos, CA, USA. IEEE Computer Society. find
  14. Lars Benedikt Kaesberg, Jonas Becker, Jan Philip Wahle, Terry Ruas, and Bela Gipp. 2025a. Voting or Consensus? Decision-Making in Multi-Agent Debate. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11640– 11671. find
  15. Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. 2024a. Mixture-of-Agents Enhances Large Language Model Capabilities. Preprint, arXiv:2406.04692.
  16. Lars Benedikt Kaesberg, Jan Philip Wahle, Terry Ruas, and Bela Gipp. 2025b. SPaRC: A spatial pathfinding reasoning challenge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 10359–10390, Suzhou, China. Association for Computational Linguistics. find
  17. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. Preprint, arXiv:2203.11171.
  18. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024b. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmar…
  19. Lars Benedikt Kaesberg, Tianyu Yang, Niklas Bauer, Terry Ruas, Jan Philip Wahle, and Bela Gipp. 2026. Mind the gap between spatial reasoning and acting! step-by-step evaluation of agents with spatial-gym. Preprint, arXiv:2604.09338. — [arXiv:2604.09338](https://arxiv.org/abs/2604.09338) find
  20. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. Preprint, arXiv:2001.08361.
  21. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Preprint, arXiv:2201.11903.
  22. Li, Linden. 2023. Transformer Inference: From First Principles to the Current State of the Art. NeurIPS Conference, New Orleans, USA. find
  23. Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. 2024. Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models. Preprint, arXiv:2408.00724.
  24. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative Refinement with… find
  25. Enhao Zhang, Erkang Zhu, Gagan Bansal, Adam Fourney, Hussein Mozannar, and Jack Gerrits. 2025a. Optimizing Sequential Multi-Step Tasks with Parallel LLM Agents. find
  26. Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Zhihan Guo, Yufei Wang, Irwin King, Xue Liu, and Chen Ma. 2025b. A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
  27. META. 2024. The Llama 3 Herd of Models. Preprint, arXiv:2407.21783.
  28. NVIDIA Corporation. 2021. NVIDIA A100 Tensor Core GPU Data Sheet. Technical report, NVIDIA. find
  29. Ranajoy Sadhukhan, Zhuoming Chen, Haizhong Zheng, Yang Zhou, Emma Strubell, and Beidi Chen. 2025. Kinetics: Rethinking Test-Time Scaling Laws. find
  30. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. Preprint, arXiv:2408.03314.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wunderlich-2026-multi-agent-reasoning-improves-compute/