Elamur Research⌕
☰
home / papers / 2026 / Hetero-Designer: Automated Design of Multi-Agent Systems with Heterogeneous LLMs

Hetero-Designer: Automated Design of Multi-Agent Systems with Heterogeneous LLMs

Paper · 2026 · Zhiheng Zhang, Yuanzhe Zhang, Bohan Yu et al.

Summary

Most automated methods for building LLM-based multi-agent systems (MAS) run every agent on the same model, such as GPT-4o. The authors argue this produces redundant topologies, little diversity in reasoning, and a poor balance between accuracy and cost. They define a new task, Automated Design of Heterogeneous-LLMs-based MAS (ADHM): for each query, jointly choose the collaboration graph, the agent roles and the LLM behind each agent, rewarding performance and penalizing API token cost.

Their pipeline, Hetero-Designer, works in three steps:

  • It encodes the query, candidate LLMs, roles and an end token with a sentence encoder.
  • A Binary-Star Transformer (BSFormer) turns these into dependency-aware representations.
  • A GRU-based graph encoder builds a directed acyclic agent graph autoregressively, alternating node and edge predictions until the end token or an agent limit is reached.

Training combines a supervised cold start with reinforcement learning. According to the authors, the method beats state-of-the-art baselines in accuracy and cost savings across several benchmarks and transfers to LLMs and roles it has not seen. This suggests that mixing cheaper and stronger models can raise the quality of collective reasoning.

Key points

  • BSFormer has two mirrored star-shaped modules, LLM-former and Role-former. In each, the query is the hub and the LLMs or roles are satellites. Multi-head attention runs inside each module, and cross-attention passes information between them.
  • The training objective maximizes expected task utility minus λ times token cost. Generation stops when the end node v_end is predicted or a maximum agent count γ is reached.
  • In the motivating MMLU example (Figure 1), a small heterogeneous MAS answers correctly where a larger homogeneous MAS fails. It uses 1601 tokens instead of 5152, which the figure reports as a 69% token saving and 87.5% cost saving.
  • t-SNE plots of BSFormer embeddings on MATH show clearly separated roles within gpt-4o-mini and distinct clusters for different LLM-role pairs.
  • Stated limitations:
    • Roles come from a fixed predefined set rather than being discovered automatically.
    • There is no long-term collaborative memory across tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China246 works · 24,951 citations🔬Institute of AutomationBeijing · China87 works · 14,883 citations🎓University of Chinese Academy of SciencesBeijing · China188 works · 8,743 citations🔬National Science LibraryBeijing · China4 works · 16 citations🎓Hunan Normal UniversityChangsha · China3 works

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Zhiheng Zhang, Yuanzhe Zhang, Bohan Yu, Daojian Zeng, Kang Liu and Jun Zhao. 2026. Hetero-Designer: Automated Design of Multi-Agent Systems with Heterogeneous LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1272

@inproceedings{zhang2026hetero,
  title = {{Hetero-Designer: Automated Design of Multi-Agent Systems with Heterogeneous LLMs}},
  author = {Zhiheng Zhang, Yuanzhe Zhang, Bohan Yu, Daojian Zeng, Kang Liu and Jun Zhao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1272},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1272}
}

References

57 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  2. Bradley P Carlin and Thomas A Louis. 2000. Empirical bayes: Past, present and future. Journal of the American Statistical Association, 95(452):1286–1289. find
  3. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research.
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — Evaluating Large Language Models Trained on Code
  5. Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang. 2024. RouterDC: Query-based router by dual contrastive learning for assembling large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  6. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. Advances in neural information processing systems. find
  7. Shengran Hu, Cong Lu, and Jeff Clune. 2025a. Automated design of agentic systems. In The Thirteenth International Conference on Learning Representations.
  8. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  9. Yuyang Hu, Shichun Liu, Yanwei Yue, Guibin Zhang, Boyang Liu, Fangyi Zhu, Jiahang Lin, Honglin Guo, Shihan Dou, Zhiheng Xi, Senjie Jin, Jiejun Tan, Yanbin Yin, Jiongnan Liu, Zeyu Zhang, Zhongxiang Sun, Yutao Zhu, Hao Sun, Boci Peng, and 28 others. 2025b. Memory in the age of ai agents. Preprint, arX… — [arXiv:2512.13564](https://arxiv.org/abs/2512.13564) find
  10. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2024. Improving factuality and reasoning in language models through multiagent debate. In International Conference on Machine Learning, pages 11733–11763. PMLR.
  11. Junkai Li, Yunghwei Lai, Weitao Li, Jingyi Ren, Meng Zhang, Xinhui Kang, Siyu Wang, Peng Li, Ya-Qin Zhang, Weizhi Ma, and Yang Liu. 2025. Agent hospital: A simulacrum of hospital with evolvable medical agents. Preprint, arXiv:2405.02957. — [arXiv:2405.02957](https://arxiv.org/abs/2405.02957) find
  12. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2025. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning.
  13. Shiyuan Li, Yixin Liu, Qingsong Wen, Chengqi Zhang, and Shirui Pan. 2026. Assemble your crew: Automatic multi-agent communication topology design via autoregressive graph generation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(28):23142–23150. find
  14. Muxin Fu, Xiangyuan Xue, Yafu Li, Zefeng He, Siyuan Huang, Xiaoye Qu, Yu Cheng, and Yang Yang. 2026. Latentmem: Customizing latent memory for multi-agent systems. Preprint, arXiv:2602.03036. — [arXiv:2602.03036](https://arxiv.org/abs/2602.03036) find
  15. Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-transformer. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1315–13… find
  16. Mahmood Hegazy. 2025. Diversity of thought elicits stronger reasoning capabilities in multi-agent debate frameworks. International Journal of Computer Science and Mobile Applications. find
  17. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. In International Conference on Learning Representations.
  18. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  19. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta programming for a multi-agent collaborative framework. In The T…
  20. Yunxuan Li, Yibing Du, Jiageng Zhang, Le Hou, Peter Grabowski, Yeqing Li, and Eugene Ie. 2024. Improving multi-agent debate with sparse communication topology. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7281– 7294, Miami, Florida, USA. Association for Computation… find
  21. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17…
  22. Yixin Liu, Guibin Zhang, Kun Wang, Shiyuan Li, Shirui Pan, and Bo An. 2026. Graph-Augmented Large Language Model Agents: Current Progress and Future Prospects . IEEE Intelligent Systems, 41(02):45– 55. find
  23. Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. 2024. A dynamic LLM-powered agent network for task-oriented agent collaboration. In First Conference on Language Modeling.
  24. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605. find
  25. Yohei Nakajima. 2023. Babyagi. GitHub repository. find
  26. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. 2025. RouteLLM: Learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations. find
  27. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, New York, NY, USA. Association f… find
  28. Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. 2025. Scaling large language model-based multi-agent collaboration. In The Thirteenth International Conference on Learning Representations.
  29. Toran Bruce Richards. 2023. Auto-gpt: An autonomous gpt-4 experiment. find
  30. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. 2025. Agent laboratory: Using LLM agents as research assistants. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 5977– 6043, Suzhou, Ch…
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems.
  32. Song Wang, Zhen Tan, Zihan Chen, Shuang Zhou, Tianlong Chen, and Jundong Li. 2025a. AnyMAC: Cascading flexible multi-agent collaboration via next-agent prediction. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 11555–11567, Suzhou, China. Association…
  33. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33:5776–5788. find
  34. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
  35. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2024. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. In Proceedings of the 2024 Conference of the North American Chapter of the Association…
  36. Zhexuan Wang, Yutong Wang, Xuebo Liu, Liang Ding, Miao Zhang, Jie Liu, and Min Zhang. 2025b. Agent-Dropout: Dynamic agent elimination for token-efficient and high-performance LLM-based multi-agent collaboration. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisti… find
  37. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
  38. Tianxin Wei, Noveen Sachdeva, Benjamin Coleman, Zhankui He, Yuanchen Bei, Xuying Ning, Mengting Ai, Yunzhe Li, Jingrui He, Ed H. Chi, Chi Wang, Shuo Chen, Fernando Pereira, Wang-Cheng Kang, and Derek Zhiyuan Cheng. 2025. Evo-memory: Benchmarking llm agent test-time learning with self-evolving memory… — [arXiv:2511.20857](https://arxiv.org/abs/2511.20857) find
  39. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2024. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language… find
  40. Jiaxi Yang, Mengqi Zhang, Yiqiao Jin, Hao Chen, Qingsong Wen, Lu Lin, Yi He, Weijie Xu, James Evans, and Jindong Wang. 2025. Topological structure learning should be a research priority for llm-based multi-agent systems. Preprint, arXiv:2505.22467. — [arXiv:2505.22467](https://arxiv.org/abs/2505.22467) find
  41. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems. find
  42. Rui Ye, Xiangrui Liu, Qimin Wu, Xianghe Pang, Zhenfei Yin, Lei Bai, and Siheng Chen. 2025. X-mas: Towards building multi-agent systems with heterogeneous llms. Preprint, arXiv:2505.16997. — [arXiv:2505.16997](https://arxiv.org/abs/2505.16997) find
  43. Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. 2018. Graphrnn: Generating realistic graphs with deep auto-regressive models. In International conference on machine learning, pages 5708–5717. PMLR. find
  44. Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Dongsheng Li, and Deqing Yang. 2025. EvoAgent: Towards automatic multi-agent generation via evolutionary algorithms. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Lan… find
  45. Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. 2025. MasRouter: Learning to route LLMs for multi-agent systems. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15549–15572, Vienna, Au…
  46. Ruijia Zhang, Xinyan Zhao, Ruixiang Wang, Sigen Chen, Guibin Zhang, An Zhang, Kun Wang, and Qingsong Wen. 2026a. Safesieve: From heuristics to experience in progressive pruning for llm-based multi-agent communication. Proceedings of the AAAI Conference on Artificial Intelligence, 40(35):29892–29900. find
  47. Guibin Zhang, Kaijie Chen, Guancheng Wan, Heng Chang, Hong Cheng, Kun Wang, Shuyue Hu, and Lei Bai. 2025a. Evoflow: Evolving diverse agentic workflows on the fly. Preprint, arXiv:2502.07373. — [arXiv:2502.07373](https://arxiv.org/abs/2502.07373) find
  48. Zhiheng Zhang, Yuanzhe Zhang, Daojian Zeng, Kang Liu, and Jun Zhao. 2026b. Beware of the woozle effect: Exploring and mitigating hallucination propagation in multi-agent debate. IEEE Transactions on Audio, Speech and Language Processing, 34:2021– 2032. find
  49. Guibin Zhang, Muxin Fu, Kun Wang, Guancheng Wan, Miao Yu, and Shuicheng YAN. 2025b. G-memory: Tracing hierarchical memory for multi-agent systems. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  50. Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, LEI BAI, and Xiang Wang. 2025c. Multi-agent architecture search via agentic supernet. In Forty-second International Conference on Machine Learning. find
  51. Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. 2025d. Cut the crap: An economical communication pipeline for LLM-based multi-agent systems. In The Thirteenth International Conference on Learning Representations. find
  52. Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, Tianlong Chen, and Dawei Cheng. 2025e. G-designer: Architecting multi-agent communication topologies via graph neural networks. In ICLR 2025 Workshop on Foundation Models in the Wild. find
  53. Hangfan Zhang, Zhiyao Cui, Jianhao Chen, Xinrun Wang, Qiaosheng Zhang, Zhen Wang, Dinghao Wu, and Shuyue Hu. 2025f. Stop overvaluing multi-agent debate – we must rethink evaluation and embrace model heterogeneity. Preprint, arXiv:2502.08788. — [arXiv:2502.08788](https://arxiv.org/abs/2502.08788) find
  54. Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, and 1 others. 2025g. Aflow: Automating agentic workflow generation. In The Thirteenth International Conference on Learning Representations.
  55. Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024. CodeAgent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1…
  56. Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Garnett, and Yixin Chen. 2019. D-vae: A variational autoencoder for directed acyclic graphs. Advances in neural information processing systems. find
  57. Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. 2024. GPTSwarm: Language agents as optimizable graphs. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 62743–…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-hetero-designer-automated-design-multi/