Elamur Research⌕
☰
home / papers / 2026 / Superficial Success vs. Internal Breakdown: An Empirical Study of Generalization in Adaptive Multi-Agent Systems

Superficial Success vs. Internal Breakdown: An Empirical Study of Generalization in Adaptive Multi-Agent Systems

Paper · 2026 · Namyeong So, Seokgyu Jang and Taeuk Kim

Summary

The paper examines whether adaptive multi-agent systems (MAS) can serve as general-purpose tools. In these systems, LLM agent roles and communication links are learned from a small task-specific training set. The authors test two representative optimizers on GPT-oss-20B, with Qwen3-30B-A3B in the appendix. AFlow builds topologies bottom-up, and AgentDropout prunes a fully connected graph top-down. Topologies are trained on one of six datasets covering legal, detective, multi-hop, science, math and commonsense reasoning, then applied unchanged to the other five.

Two problems emerge. The first is topological overfitting: accuracy often falls on unseen domains. The second is illusory coordination: some transfers look accurate, but agents abandon their roles or ignore each other's messages, so the answer comes from individual LLM strength rather than collaboration. To measure this, the authors introduce two metrics, Role Alignment (R) and Connection Significance (O), alongside a MAST-based failure analysis. The work matters because final-answer accuracy alone can hide broken internal collaboration. This motivates evaluation protocols that inspect agent interactions and design choices aimed at generalization, such as multi-domain training.

Key points

  • Datasets: CaseHOLD, COM², MuSiQue, SciBench, TheoremQA and StrategyQA. Training was capped at 100 examples for AFlow and 60 for AgentDropout, with AgentInit used to set AgentDropout's roles.
  • Out-of-domain drop: AgentDropout trained on CaseHOLD falls from 63.5% in-domain to about 55.8% average OOD. StrategyQA-trained (true/false) topologies collapse on numerical and multiple-choice formats. Mixing all six domains in training gives more stable results.
  • Failure analysis: Using MAST, a GPT-oss-120B judge labeled 100 execution logs. Role- and connection-related failures make up roughly 60% of breakdowns, with disobeying the role specification the largest single category (about 15%).
  • Accuracy vs. collaboration: CaseHOLD→SciBench appears to transfer well, yet R drops to 22% of its in-domain value. Pearson correlations between accuracy and R or O are close to zero.
  • Ablation: Replacing roles with OOD ones costs about 13.0 pp on average, versus about 1.24 pp for replacing connections. MuSiQue is the exception, losing 5.36 pp when connections are swapped.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hanyang UniversitySeoul · South Korea6 works · 150 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

Cite

Cite as Namyeong So, Seokgyu Jang and Taeuk Kim. 2026. Superficial Success vs. Internal Breakdown: An Empirical Study of Generalization in Adaptive Multi-Agent Systems. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.753

@inproceedings{so2026superficial,
  title = {{Superficial Success vs. Internal Breakdown: An Empirical Study of Generalization in Adaptive Multi-Agent Systems}},
  author = {Namyeong So, Seokgyu Jang and Taeuk Kim},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.753},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.753}
}

References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, and 1 others. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925.
  2. Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, and 1 others. 2025. Why do multi-agent llm systems fail? arXiv preprint arXiv:2503.13657. find
  3. Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. Lexglue: A benchmark dataset for legal language understanding in english. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 4310…
  4. Ding Chen, Qingchen Yu, Pengyuan Wang, Wentao Zhang, Bo Tang, Feiyu Xiong, Xinchi Li, Minchuan Yang, and Zhiyu Li. 2025. xverify: Efficient answer verifier for reasoning model evaluations. arXiv preprint arXiv:2504.10481. — [arXiv:2504.10481](https://arxiv.org/abs/2504.10481) find
  5. Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2024. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelft…
  6. Wenhu Chen and 1 others. 2023. Theoremqa: A theorem-driven question answering dataset. arXiv preprint arXiv:2305.12524.
  7. Mor Geva and 1 others. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361. find
  8. Google. 2025. Gemini 3 flash model card. https://storage.googleapis.com/deepmind-media/ Model-Cards/Gemini-3-Flash-Model-Card. pdf. Accessed: 2025-12-30. ↗
  9. Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with more persuasive llms leads to more truthful answers. arXiv preprint arXiv:2402.06782. find
  10. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626.
  11. Shiyuan Li, Yixin Liu, Qingsong Wen, Chengqi Zhang, and Shirui Pan. 2025. Assemble your crew: Automatic multi-agent communication topology design via autoregressive graph generation. arXiv preprint arXiv:2507.18224. — [arXiv:2507.18224](https://arxiv.org/abs/2507.18224) find
  12. OpenAI. 2025. Gpt-5.1 instant and gpt-5.1 thinking system card addendum. https://cdn.openai.com/ pdf/4173ec8d-1229-47db-96de-06d87147e07e/ 5_1_system_card.pdf. Accessed: 2025-12-30. ↗
  13. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ChatDev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computat…
  14. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  15. Chunhao Tian, Yutong Wang, Xuebo Liu, Zhexuan Wang, Liang Ding, Miao Zhang, and Min Zhang. 2025. Agentinit: Initializing llm-based multi-agent systems via diversity and expertise orchestration for effective and efficient collaboration. In Findings of the Association for Computational Linguistics: EM… find
  16. Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. 2024a. Cut the crap: An economical communication pipeline for llm-based multi-agent systems. arXiv preprint arXiv:2410.02506. — [arXiv:2410.02506](https://arxiv.org/abs/2410.02506) find
  17. Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, Tianlong Chen, and Dawei Cheng. 2024b. G-designer: Architecting multi-agent communication topologies via graph neural networks. arXiv preprint arXiv:2410.11782. — [arXiv:2410.11782](https://arxiv.org/abs/2410.11782) find
  18. Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, and 1 others. 2024c. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762. find
  19. Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jurgen Schmidhuber. 2024. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning. find
  20. Harsh Trivedi and 1 others. 2022. Musique: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554. find
  21. Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. 2023. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint arXiv:2307.10635.
  22. Zhexuan Wang, Yutong Wang, Xuebo Liu, Liang Ding, Miao Zhang, Jie Liu, and Min Zhang. 2025. Agentdropout: Dynamic agent elimination for token-efficient and high-performance llm-based multi-agent collaboration. arXiv preprint arXiv:2503.18891.
  23. Kai Xiong, Xiao Ding, Yixin Cao, Yuxiong Yan, Li Du, Yufei Zhang, Jinglong Gao, Jiaqian Liu, Bing Qin, and Ting Liu. 2025. Com2: A causal-guided benchmark for exploring complex commonsense reasoning in large language models. arXiv preprint arXiv:2506.07064. — [arXiv:2506.07064](https://arxiv.org/abs/2506.07064) find
  24. Yangyang Yu, Zhiyuan Yao, Haohang Li, Zhiyang Deng, Yuechen Jiang, Yupeng Cao, Zhi Chen, Jordan Suchow, Zhenyu Cui, Rong Liu, and 1 others. 2024. Fincon: A synthesized llm multi-agent system with conceptual verbal reinforcement for enhanced financial decision making. Advances in Neural Information P… find
  25. Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang. 2025. Multi-agent architecture search via agentic supernet. arXiv preprint arXiv:2502.04180. — [arXiv:2502.04180](https://arxiv.org/abs/2502.04180) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/so-2026-superficial-success-vs-internal-breakdown/