Elamur Research⌕
☰
home / papers / 2026 / Building LLMs Like LEGO: Two-dimensional Architecture Reassembly of Large Language Models

Building LLMs Like LEGO: Two-dimensional Architecture Reassembly of Large Language Models

Paper · 2026 · Xingyu Wu, Yu Zhou and KC Tan

Summary

Pretrained LLMs are usually reused as whole units: they are fine-tuned, merged at the weight level, or ensembled at the output. The authors propose LEGO-LLM, which breaks heterogeneous models into Transformer blocks and assembles new architectures from them without the original training data. The search space has two dimensions. Blocks can be stacked across depth (vertical), or several models can contribute to a single layer (horizontal). Each candidate design is encoded as a chromosome that records which blocks are chosen, the operator applied at each depth, and the alignment layers between them. A bilevel multi-objective evolutionary algorithm explores these candidates. Small glue layers between blocks resolve mismatches in hidden size and representation. They are trained by data-free knowledge distillation, and the source weights stay frozen. On cross-lingual and cross-cultural benchmarks (Mela, XNLI, SeaEval, Flores), the reassembled models outperform parameter-space reuse methods, and on an Arabic task they beat every source model. The work presents architecture recombination as a third reuse path alongside fine-tuning and merging.

Key points

  • Four layer-level assembly operators are defined:
    • vertical substitution (stitching)
    • parameter-level merging
    • horizontal concatenation (layer widening)
    • output-level ensemble
  • In the glue-layer ablation, the default setup trains glue layers on synthetic Gaussian inputs. It scores highest on Mela, XNLI, SeaEval and Flores (0.3200 / 0.5163 / 0.5940 / 0.3460). The alternatives score lower:
    • Using 50% of XNLI real tokens gives 0.3024 / 0.4868 / 0.5631 / 0.3272.
    • Replacing glue layers with a plain MLP projection gives 0.2423 / 0.4089 / 0.5068 / 0.2694.
  • Reported search cost is roughly 1.25–6.25 h for evolutionary reassembly versus 2.00–6.66 h for fine-tuning and about 0.095 h for parameter merging. Peak GPU memory is roughly 20.5–26.5 GB for reassembly versus 41–43 GB for fine-tuning.
  • Candidates are evaluated with forward-only inference, and only the glue modules are trained, so the search is parallelizable across the population.
  • The authors list two limitations: higher performance variance than with fine-tuning, and search overhead that grows with larger search spaces. As future work, they plan to harmonize normalization, positional encoding, attention masking and residual conventions across fully heterogeneous models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong Polytechnic UniversityHong Kong · China84 works · 4,956 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Xingyu Wu, Yu Zhou and KC Tan. 2026. Building LLMs Like LEGO: Two-dimensional Architecture Reassembly of Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2081

@inproceedings{wu2026building,
  title = {{Building LLMs Like LEGO: Two-dimensional Architecture Reassembly of Large Language Models}},
  author = {Xingyu Wu, Yu Zhou and KC Tan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2081},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2081}
}

References

43 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. 2025. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 7(2):195–204. find
  2. Pin-Yu Chen. 2024. Model reprogramming: Resource-efficient cross-domain machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22584–22591. find
  3. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485. find
  4. Xingyu Dang, Christina Baek, Kaiyue Wen, Zico Kolter, and Aditi Raghunathan. 2025. Weight ensembling improves reasoning in language models. arXiv preprint arXiv:2504.10478. find
  5. Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Transactions on Evolutionary Computation, 6(2):182–197. find
  6. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. find
  7. Shangbin Feng, Wenxuan Ding, Alisa Liu, Zifeng Wang, Weijia Shi, Yike Wang, Zejiang Shen, Xiaochuang Han, Hunter Lang, Chen-Yu Lee, and 1 others. 2025. When one llm drools, multi-llm collaboration rules. arXiv preprint arXiv:2502.04506.
  8. Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. Arcee’s mergekit: A toolkit for merging large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing:…
  9. Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789–1819. find
  10. Neel Guha, Mayee F Chen, Trevor Chow, Ishan S Khare, and Christopher Ré. 2024. Smoothie: label free language model routing. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pages 127645–127672. find
  11. Lan-Zhe Guo, Zhi Zhou, Yu-Feng Li, and Zhi-Hua Zhou. 2023. Identifying useful learnwares for heterogeneous label spaces. In Proceedings of the International Conference on Machine Learning, pages 12122–12131. PMLR. find
  12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In Proceedings of the 9th International Conference on Learning Representations.
  13. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In Proceedings of the 36th International conference on machine learning, pages 2790–2799. PMLR. find
  14. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. In Proceedings of the 10th International Conference on Learning Representations.
  15. Shaohan Huang, Yi Liu, Carol Fung, Jiaxing Qi, Hailong Yang, and Zhongzhi Luan. 2023. Logqa: Question answering in unstructured logs. arXiv preprint arXiv:2303.11715. — [arXiv:2303.11715](https://arxiv.org/abs/2303.11715) find
  16. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87. find
  17. Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat, Jeevesh Juneja, Zifeng Wang, Chen-Yu Lee, Pradeep Shenoy, Rina Panigrahy, Aditya Krishna Menon, and Sanjiv Kumar. 2025. Universal llm routing with correctness-based representation. In Proceedings of the First Workshop on Scalable Optimi… find
  18. Akhil Kedia, Mohd Abbas Zaidi, Sushil Khyalia, Jungho Jung, Harshith Goka, and Haejun Lee. 2024. Transformers get stable: An end-to-end signal propagation theory for language models. In Proceedings of the 41st International Conference on Machine Learning, pages 23449–23531. PMLR. find
  19. Sanghoon Kim, Dahyun Kim, Chanjun Park, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, and 1 others. 2024. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. In Proceedings of the 2024 Conference of the North American Chapter… find
  20. Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2024. Routing to the expert: Efficient reward-guided ensemble of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Huma… find
  21. Michael Matena and Colin Raffel. 2022. Merging models with fisher-weighted averaging. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 17703–17716.
  22. Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Reuse, don’t retrain: A recipe for continued pre-training of language models. arXiv preprint arXiv:2407.07263.
  23. Jonas Pfeiffer, Sebastian Ruder, Ivan Vulic, and Edoardo M Ponti. 2023. Modular deep learning. Transation on Machine Learning Research. find
  24. Haiyun Qiu, Xingyu Wu, Liang Feng, and Kay Chen Tan. 2026. Fine-grained model merging via modular expert recombination. arXiv preprint arXiv:2602.06552. — [arXiv:2602.06552](https://arxiv.org/abs/2602.06552) find
  25. Chongjie Si, Jingjing Jiang, and Wei Shen. 2025. Unveiling the mystery of weight in large foundation models: Gaussian distribution never fades. arXiv preprint arXiv:2501.10661. — [arXiv:2501.10661](https://arxiv.org/abs/2501.10661) find
  26. Sidak Pal Singh and Martin Jaggi. 2020. Model fusion via optimal transport. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 22045–22055. find
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Proceedings of the 31st Annual Conference on Advances in neural information processing systems, 30.
  28. Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. Knowledge fusion of large language models. In Proceedings of the 12th International Conference on Learning Representations. find
  29. Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, Aiti Aw, and Nancy Chen. 2024a. Seaeval for multilingual foundation models: From cross-lingual alignment to cultural reasoning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingui…
  30. Junlin Wang, WANG Jue, Ben Athiwaratkun, Ce Zhang, and James Zou. 2024b. Mixture-of-agents enhances large language model capabilities. In Proceedings of the 13th International Conference on Learning Representations. find
  31. Qiu-Feng Wang, Xin Geng, Shu-Xia Lin, Shi-Yu Xia, Lei Qi, and Ning Xu. 2022. Learngene: From open-world to your learning task. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8557–8565. find
  32. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and 1 others. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
  33. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and 1 others. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time…
  34. Xingyu Wu, Jibin Wu, Yu Zhou, Liang Feng, and Kay Chen Tan. 2025. Towards robustness and explainability of automatic algorithm selection. In Proceedings of the 42nd International Conference on Machine Learning, pages 67864–67887. PMLR. find
  35. Xingyu Wu, Sheng-hao Wu, Jibin Wu, Liang Feng, and Kay Chen Tan. 2024. Evolutionary computation in the era of large language model: Survey and roadmap. IEEE Transactions on Evolutionary Computation, 29(2):534–554. find
  36. Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. 2024. Ties-merging: resolving interference when merging models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 7093–7115.
  37. Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. 2024. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666. find
  38. Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. 2022. Deep model reassembly. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 25739–25753. find
  39. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: absorbing abilities from homologous models as a free lunch. In Proceedings of the 41st International Conference on Machine Learning, pages 57755–57775.
  40. Runjia Zeng, James Chenhao Liang, Cheng Han, Zhiwen Cao, Jiahao Liu, Xiaojun Quan, Yingjie Victor Chen, Lifu Huang, Tong Geng, Qifan Wang, and 1 others. 2025. Probabilistic token alignment for large language model fusion. arXiv preprint arXiv:2509.17276. — [arXiv:2509.17276](https://arxiv.org/abs/2509.17276) find
  41. Ziyin Zhang, Yikang Liu, Weifang Huang, Junyu Mao, Rui Wang, and Hai Hu. 2023. Mela: Multilingual evaluation of linguistic acceptability. arXiv preprint arXiv:2311.09033. find
  42. Xun Zhou, Xingyu Wu, Liang Feng, Zhichao Lu, and Kay Chen Tan. 2025a. Design principle transfer in neural architecture search via large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23000–23008. find
  43. Yu Zhou, Xingyu Wu, Jibin Wu, Liang Feng, and KC Tan. 2025b. Hm3: Hierarchical multi-objective model merging for pretrained models. In Proceedings of the 39th Annual Conference on Neural Information Processing Systems. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-building-llms-like-lego-two/